From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AA6B8A0545 for ; Fri, 11 Nov 2022 12:41:02 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A5A9F427EB; Fri, 11 Nov 2022 12:41:02 +0100 (CET) Received: from smtp-relay-internal-1.canonical.com (smtp-relay-internal-1.canonical.com [185.125.188.123]) by mails.dpdk.org (Postfix) with ESMTP id 3822940141 for ; Fri, 11 Nov 2022 12:41:01 +0100 (CET) Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 058613F0C8 for ; Fri, 11 Nov 2022 11:41:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1668166861; bh=6RJzNtuQYa5xFGmTWKZsBh71LC07+wyqMXJLSEKllAk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=p6O0Gs7MjXyJFGmR9ohgrxuvQyjLn5evtCuFKy6TusU6XRcuNb4skDnydtJ+DmVJ3 tyiJpSEHdoCIh50LiDVpRpzE/RiTu9K+DFTDEilJeisB0z37uaD82A3Kz8ww8v9yfq Oirrxy6ctycWY71mIga3Yi3cFbuSA8sfqBeba38X/+774GJ9Z0h/6crpEHMqj/vV8G b5MryqogI1+Hg566sFhRsXscMzOZ/488/jQDkjJcsL7Rr12coS5ttveu7SYb0zKEPj lJh3Ut2UQpcDGbbkQP7eYd3xJj3zHaE/tPgaH/6m/rlNHdV7L1ScKJMWQm22thuega KO6x6YgD/7tkA== Received: by mail-ej1-f69.google.com with SMTP id jg27-20020a170907971b00b007ad9892f5f6so2895609ejc.7 for ; Fri, 11 Nov 2022 03:41:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6RJzNtuQYa5xFGmTWKZsBh71LC07+wyqMXJLSEKllAk=; b=eGEbimvz13AcxIdX7DuZtlSnXEZi81W62zICqR1S4DmlVM4SAvY1TQanrtzb8sT1c2 3bykAkG4EzZrFjZiTRptNgqxFyhiLKxkQHZgyAcC2o0K9j3eSvnI46GEath8eMhNERYe mUBQLHJ8LWK30wTHToe/7lcUQ9Mb4NdpPw1bIe26vBLizU+2jyqYfOcYoW3esPCoCUCw e2CaONKZbZlE/uHRO2CoyutpqG5Jlx1MKBj7TqVT/u1dr/gquZNAGueVi+UaXifaaZX3 f4KAKo+0abe+Wsrs/IQQXhI+EHdtL5Khk1y/Pyi66NnUEPBcPWg55Gqjl3+3vTRyVwvs 7QDA== X-Gm-Message-State: ANoB5plHgGO0k6tetNxgNqgL5x2QfgisDflAhf9j/LpX1xA3Qxnl84UK UrG9crfPGDt5pD56lyWFEaJfUGP73dKdj3sVhW186lwF03b/9aQTIPnsqp5YbNULJUIl/bB+vI7 dVqaKgKuj6BrGJ9bM0kmXvZ+q X-Received: by 2002:a17:907:76b8:b0:7ae:6cdd:9bc9 with SMTP id jw24-20020a17090776b800b007ae6cdd9bc9mr1419267ejc.619.1668166860505; Fri, 11 Nov 2022 03:41:00 -0800 (PST) X-Google-Smtp-Source: AA0mqf6zYs6dmsRFXU4rn9f5TIVDUsakbUI/BwhmGoOhebYNCLvNh/U29OXw7RaNmOy208kTPOQiuA== X-Received: by 2002:a17:907:76b8:b0:7ae:6cdd:9bc9 with SMTP id jw24-20020a17090776b800b007ae6cdd9bc9mr1419253ejc.619.1668166860299; Fri, 11 Nov 2022 03:41:00 -0800 (PST) Received: from localhost.localdomain ([2a02:6d40:39af:c100:1891:6c61:50fd:e567]) by smtp.gmail.com with ESMTPSA id jz22-20020a17090775f600b0073dc4385d3bsm782322ejc.105.2022.11.11.03.40.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Nov 2022 03:40:59 -0800 (PST) From: christian.ehrhardt@canonical.com To: Stephen Hemminger Cc: Anatoly Burakov , Chengwen Feng , dpdk stable Subject: patch 'eal: fix data race in multi-process support' has been queued to stable release 19.11.14 Date: Fri, 11 Nov 2022 12:38:52 +0100 Message-Id: <20221111113904.1549618-35-christian.ehrhardt@canonical.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221111113904.1549618-1-christian.ehrhardt@canonical.com> References: <20221111113904.1549618-1-christian.ehrhardt@canonical.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 19.11.14 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/18/22. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/cpaelzer/dpdk-stable-queue This queued commit can be viewed at: https://github.com/cpaelzer/dpdk-stable-queue/commit/910171a808f8c5d2e709b5aa2aaba09179a85265 Thanks. Christian Ehrhardt --- >From 910171a808f8c5d2e709b5aa2aaba09179a85265 Mon Sep 17 00:00:00 2001 From: Stephen Hemminger Date: Tue, 6 Sep 2022 09:45:22 -0700 Subject: [PATCH] eal: fix data race in multi-process support [ upstream commit 668958f3c1617f18e04ffee099656e7fb2effa94 ] If DPDK is built with thread sanitizer it reports a race in setting of multiprocess file descriptor. The fix is to use atomic operations when updating mp_fd. Build: $ meson -Db_sanitize=address build $ ninja -C build Simple example: $ .build/app/dpdk-testpmd -l 1-3 --no-huge EAL: Detected CPU lcores: 16 EAL: Detected NUMA nodes: 1 EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem EAL: Detected static linkage of DPDK EAL: Multi-process socket /run/user/1000/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' testpmd: No probed ethernet devices testpmd: create a new mbuf pool : n=163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc EAL: Error - exiting with code: 1 Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory ================== WARNING: ThreadSanitizer: data race (pid=87245) Write of size 4 at 0x558e04d8ff70 by main thread: #0 rte_mp_channel_cleanup (dpdk-testpmd+0x1e7d30c) #1 rte_eal_cleanup (dpdk-testpmd+0x1e85929) #2 rte_exit (dpdk-testpmd+0x1e5bc0a) #3 mbuf_pool_create.cold (dpdk-testpmd+0x274011) #4 main (dpdk-testpmd+0x5cc15d) Previous read of size 4 at 0x558e04d8ff70 by thread T2: #0 mp_handle (dpdk-testpmd+0x1e7c439) #1 ctrl_thread_init (dpdk-testpmd+0x1e6ee1e) As if synchronized via sleep: #0 nanosleep libsanitizer/tsan/tsan_interceptors_posix.cpp:366 #1 get_tsc_freq (dpdk-testpmd+0x1e92ff9) #2 set_tsc_freq (dpdk-testpmd+0x1e6f2fc) #3 rte_eal_timer_init (dpdk-testpmd+0x1e931a4) #4 rte_eal_init.cold (dpdk-testpmd+0x29e578) #5 main (dpdk-testpmd+0x5cbc45) Location is global 'mp_fd' of size 4 at 0x558e04d8ff70 (dpdk-testpmd+0x000003122f70) Thread T2 'rte_mp_handle' (tid=87248, running) created by main thread at: #0 pthread_create libsanitizer/tsan/tsan_interceptors_posix.cpp:969 #1 rte_ctrl_thread_create (dpdk-testpmd+0x1e6efd0) #2 rte_mp_channel_init.cold (dpdk-testpmd+0x29cb7c) #3 rte_eal_init (dpdk-testpmd+0x1e8662e) #4 main (dpdk-testpmd+0x5cbc45) SUMMARY: ThreadSanitizer: data race (app/dpdk-testpmd+0x1e7d30c) in rte_mp_channel_cleanup ================== ThreadSanitizer: reported 1 warnings Fixes: bacaa2754017 ("eal: add channel for multi-process communication") Signed-off-by: Stephen Hemminger Acked-by: Anatoly Burakov Reviewed-by: Chengwen Feng --- lib/librte_eal/common/eal_common_proc.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c index d88a1ebafb..a98d67101d 100644 --- a/lib/librte_eal/common/eal_common_proc.c +++ b/lib/librte_eal/common/eal_common_proc.c @@ -257,7 +257,7 @@ rte_mp_action_unregister(const char *name) } static int -read_msg(struct mp_msg_internal *m, struct sockaddr_un *s) +read_msg(int fd, struct mp_msg_internal *m, struct sockaddr_un *s) { int msglen; struct iovec iov; @@ -278,7 +278,7 @@ read_msg(struct mp_msg_internal *m, struct sockaddr_un *s) msgh.msg_controllen = sizeof(control); retry: - msglen = recvmsg(mp_fd, &msgh, 0); + msglen = recvmsg(fd, &msgh, 0); /* zero length message means socket was closed */ if (msglen == 0) @@ -385,11 +385,12 @@ mp_handle(void *arg __rte_unused) { struct mp_msg_internal msg; struct sockaddr_un sa; + int fd; - while (mp_fd >= 0) { + while ((fd = __atomic_load_n(&mp_fd, __ATOMIC_RELAXED)) >= 0) { int ret; - ret = read_msg(&msg, &sa); + ret = read_msg(fd, &msg, &sa); if (ret <= 0) break; @@ -631,9 +632,8 @@ rte_mp_channel_init(void) NULL, mp_handle, NULL) < 0) { RTE_LOG(ERR, EAL, "failed to create mp thread: %s\n", strerror(errno)); - close(mp_fd); close(dir_fd); - mp_fd = -1; + close(__atomic_exchange_n(&mp_fd, -1, __ATOMIC_RELAXED)); return -1; } @@ -649,11 +649,10 @@ rte_mp_channel_cleanup(void) { int fd; - if (mp_fd < 0) + fd = __atomic_exchange_n(&mp_fd, -1, __ATOMIC_RELAXED); + if (fd < 0) return; - fd = mp_fd; - mp_fd = -1; pthread_cancel(mp_handle_tid); pthread_join(mp_handle_tid, NULL); close_socket_fd(fd); -- 2.38.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2022-11-11 12:35:06.020999734 +0100 +++ 0035-eal-fix-data-race-in-multi-process-support.patch 2022-11-11 12:35:04.793192173 +0100 @@ -1 +1 @@ -From 668958f3c1617f18e04ffee099656e7fb2effa94 Mon Sep 17 00:00:00 2001 +From 910171a808f8c5d2e709b5aa2aaba09179a85265 Mon Sep 17 00:00:00 2001 @@ -5,0 +6,2 @@ +[ upstream commit 668958f3c1617f18e04ffee099656e7fb2effa94 ] + @@ -62 +63,0 @@ -Cc: stable@dpdk.org @@ -68 +69 @@ - lib/eal/common/eal_common_proc.c | 17 ++++++++--------- + lib/librte_eal/common/eal_common_proc.c | 17 ++++++++--------- @@ -71,5 +72,5 @@ -diff --git a/lib/eal/common/eal_common_proc.c b/lib/eal/common/eal_common_proc.c -index 313060528f..1fc1d6c53b 100644 ---- a/lib/eal/common/eal_common_proc.c -+++ b/lib/eal/common/eal_common_proc.c -@@ -260,7 +260,7 @@ rte_mp_action_unregister(const char *name) +diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c +index d88a1ebafb..a98d67101d 100644 +--- a/lib/librte_eal/common/eal_common_proc.c ++++ b/lib/librte_eal/common/eal_common_proc.c +@@ -257,7 +257,7 @@ rte_mp_action_unregister(const char *name) @@ -84 +85 @@ -@@ -281,7 +281,7 @@ read_msg(struct mp_msg_internal *m, struct sockaddr_un *s) +@@ -278,7 +278,7 @@ read_msg(struct mp_msg_internal *m, struct sockaddr_un *s) @@ -93 +94 @@ -@@ -390,11 +390,12 @@ mp_handle(void *arg __rte_unused) +@@ -385,11 +385,12 @@ mp_handle(void *arg __rte_unused) @@ -108 +109 @@ -@@ -638,9 +639,8 @@ rte_mp_channel_init(void) +@@ -631,9 +632,8 @@ rte_mp_channel_init(void) @@ -119 +120 @@ -@@ -656,11 +656,10 @@ rte_mp_channel_cleanup(void) +@@ -649,11 +649,10 @@ rte_mp_channel_cleanup(void)