From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 89094430DB; Wed, 23 Aug 2023 16:02:25 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 77110410F1; Wed, 23 Aug 2023 16:02:25 +0200 (CEST) Received: from inbox.dpdk.org (inbox.dpdk.org [95.142.172.178]) by mails.dpdk.org (Postfix) with ESMTP id C176F40223 for ; Wed, 23 Aug 2023 16:02:23 +0200 (CEST) Received: by inbox.dpdk.org (Postfix, from userid 33) id 99F39430E2; Wed, 23 Aug 2023 16:02:23 +0200 (CEST) From: bugzilla@dpdk.org To: dev@dpdk.org Subject: [Bug 1277] memory_hotplug_lock deadlock during initialization Date: Wed, 23 Aug 2023 14:02:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: DPDK X-Bugzilla-Component: core X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: artemyko@nvidia.com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: dev@dpdk.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: multipart/alternative; boundary=16927993430.bCf0221a5.1434830 Content-Transfer-Encoding: 7bit X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --16927993430.bCf0221a5.1434830 Date: Wed, 23 Aug 2023 16:02:23 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All https://bugs.dpdk.org/show_bug.cgi?id=3D1277 Bug ID: 1277 Summary: memory_hotplug_lock deadlock during initialization Product: DPDK Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: Normal Component: core Assignee: dev@dpdk.org Reporter: artemyko@nvidia.com Target Milestone: --- It seems the issue arose due to changes in the DPDK read-write lock implementation. Following these changes, the RW-lock no longer supports recursion, implying that a single thread shouldn't obtain a read lock if it already possesses one. The problem arises during initialization: the rte_eal_memory_init() function acquires the memory_hotplug_lock, and later = on, the sequence of calls eal_memalloc_init() -> rte_memseg_list_walk() acquire= s it again without releasing it. This scenario introduces the risk of a potential deadlock when concurrent write locks are applied to the same memory_hotplug_lock. To address this locally, we resolved the issue by replacing rte_memseg_list_walk() with rte_memseg_list_walk_thread_unsafe(). Reproduction: Create mp_deadlock directory under dpdk/examples/. Then add main.c /* SPDX-License-Identifier: BSD-3-Clause * Copyright(c) 2010-2014 Intel Corporation */ #include #include #include #include #include #include #include #include #include #include #include #include /* Initialization of Environment Abstraction Layer (EAL). 8< */ int main(int argc, char **argv) { int ret; ret =3D rte_eal_init(argc, argv); if (ret < 0) rte_panic("Cannot init EAL\n"); /* >8 End of initialization of Environment Abstraction Layer */ if (rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY) getchar(); else { if (rte_lcore_id() <=3D 1) { int i =3D 0; void *p; while (1) { p =3D rte_malloc_socket(NULL, 0x1000000, 0x= 1000, -1); rte_free(p); printf("malloc %d times\n", i++); } } } /* clean up the EAL */ rte_eal_cleanup(); return 0; } Compile: I followed https://doc.dpdk.org/guides/prog_guide/build_app.html a= nd some tips from related web page. Run primary: ./examples/mp_deadlock/build/mp_deadlock -l 0 --file-prefix=3D= dpdk1 --proc-type=3Dprimary Run secondary 1: ./examples/mp_deadlock/build/mp_deadlock -l 1 --file-prefix=3Ddpdk1 --proc-type=3Dsecondary Run secondary 2:=20 while true do ./examples/mp_deadlock/build/mp_deadlock -l 2 --file-prefix=3Ddpdk1 --proc-type=3Dsecondary done Stack trace. It looks like the following caused deadlock. #0 0x00007f850e97a3f2 in rte_mcfg_mem_write_lock () from /usr/local/lib64/librte_eal.so.23 And #0 0x00007f3f591b5362 in rte_mcfg_mem_read_lock () from /usr/local/lib64/librte_eal.so.23 [root@fedora dpdk]# ps -ef | grep deadlock root 7328 1004 0 20:47 pts/0 00:00:00 bash ./mp_deadlock1.sh root 7329 7328 4 20:47 pts/0 00:00:01 ./examples/mp_deadlock/build/mp_deadlock -l 0 --file-prefix=3Ddpdk1 --proc-type=3Dprimary root 7333 5693 0 20:47 pts/4 00:00:00 bash ./mp_deadlock2.sh root 7334 7333 94 20:47 pts/4 00:00:31 ./examples/mp_deadlock/build/mp_deadlock -l 1 --file-prefix=3Ddpdk1 --proc-type=3Dsecondary root 7337 5267 0 20:47 pts/1 00:00:00 bash ./mp_deadlock.sh root 7338 7337 98 20:47 pts/1 00:00:29 ./examples/mp_deadlock/build/mp_deadlock -l 2 --file-prefix=3Ddpdk1 --proc-type=3Dsecondary root 7342 5480 0 20:47 pts/2 00:00:00 grep --color=3Dauto dea= dlock [root@fedora dpdk]# pstack 7329 Thread 4 (Thread 0x7f20ae487640 (LWP 7332) "telemetry-v2"): #0 0x00007f20b200ae6f in accept () from /lib64/libc.so.6 #1 0x00007f20b1e004a3 in socket_listener () from /usr/local/lib64/librte_telemetry.so.23 #2 0x00007f20b1f85b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f20b200a6a0 in clone3 () from /lib64/libc.so.6 Thread 3 (Thread 0x7f20aec88640 (LWP 7331) "rte_mp_handle"): #0 0x00007f20b200b23d in recvmsg () from /lib64/libc.so.6 #1 0x00007f20b2137ecf in mp_handle () from /usr/local/lib64/librte_eal.so.= 23 #2 0x00007f20b1f85b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f20b200a6a0 in clone3 () from /lib64/libc.so.6 Thread 2 (Thread 0x7f20af489640 (LWP 7330) "eal-intr-thread"): #0 0x00007f20b2009c7e in epoll_wait () from /lib64/libc.so.6 #1 0x00007f20b2141c54 in eal_intr_thread_main () from /usr/local/lib64/librte_eal.so.23 #2 0x00007f20b1f85b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f20b200a6a0 in clone3 () from /lib64/libc.so.6 Thread 1 (Thread 0x7f20b1df9900 (LWP 7329) "mp_deadlock"): #0 0x00007f20b1ff984c in read () from /lib64/libc.so.6 #1 0x00007f20b1f7e914 in __GI__IO_file_underflow () from /lib64/libc.so.6 #2 0x00007f20b1f7f946 in _IO_default_uflow () from /lib64/libc.so.6 #3 0x00007f20b1f7a328 in getc () from /lib64/libc.so.6 #4 0x000000000040113e in main () [root@fedora dpdk]# pstack 7334 Thread 3 (Thread 0x7f850b4da640 (LWP 7336) "rte_mp_handle"): #0 0x00007f850e85d23d in recvmsg () from /lib64/libc.so.6 #1 0x00007f850e989ecf in mp_handle () from /usr/local/lib64/librte_eal.so.= 23 #2 0x00007f850e7d7b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f850e85c6a0 in clone3 () from /lib64/libc.so.6 Thread 2 (Thread 0x7f850bcdb640 (LWP 7335) "eal-intr-thread"): #0 0x00007f850e85bc7e in epoll_wait () from /lib64/libc.so.6 #1 0x00007f850e993c54 in eal_intr_thread_main () from /usr/local/lib64/librte_eal.so.23 #2 0x00007f850e7d7b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f850e85c6a0 in clone3 () from /lib64/libc.so.6 Thread 1 (Thread 0x7f850e64b900 (LWP 7334) "mp_deadlock"): #0 0x00007f850e97a3f2 in rte_mcfg_mem_write_lock () from /usr/local/lib64/librte_eal.so.23 #1 0x00007f850e984509 in malloc_heap_free () from /usr/local/lib64/librte_eal.so.23 #2 0x00007f850e98508f in rte_free () from /usr/local/lib64/librte_eal.so.23 #3 0x0000000000401126 in main () [root@fedora dpdk]# pstack 7338 Thread 3 (Thread 0x7f3f55d15640 (LWP 7340) "rte_mp_handle"): #0 0x00007f3f5909823d in recvmsg () from /lib64/libc.so.6 #1 0x00007f3f591c4ecf in mp_handle () from /usr/local/lib64/librte_eal.so.= 23 #2 0x00007f3f59012b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f3f590976a0 in clone3 () from /lib64/libc.so.6 Thread 2 (Thread 0x7f3f56516640 (LWP 7339) "eal-intr-thread"): #0 0x00007f3f59096c7e in epoll_wait () from /lib64/libc.so.6 #1 0x00007f3f591cec54 in eal_intr_thread_main () from /usr/local/lib64/librte_eal.so.23 #2 0x00007f3f59012b17 in start_thread () from /lib64/libc.so.6 #3 0x00007f3f590976a0 in clone3 () from /lib64/libc.so.6 Thread 1 (Thread 0x7f3f58e86900 (LWP 7338) "mp_deadlock"): #0 0x00007f3f591b5362 in rte_mcfg_mem_read_lock () from /usr/local/lib64/librte_eal.so.23 #1 0x00007f3f591b6bf2 in rte_memseg_list_walk () from /usr/local/lib64/librte_eal.so.23 #2 0x00007f3f591d2f65 in eal_memalloc_init () from /usr/local/lib64/librte_eal.so.23 #3 0x00007f3f591b741b in rte_eal_memory_init () from /usr/local/lib64/librte_eal.so.23 #4 0x00007f3f591aab64 in rte_eal_init.cold () from /usr/local/lib64/librte_eal.so.23 #5 0x00000000004010d9 in main () --=20 You are receiving this mail because: You are the assignee for the bug.= --16927993430.bCf0221a5.1434830 Date: Wed, 23 Aug 2023 16:02:23 +0200 MIME-Version: 1.0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All
Bug ID 1277
Summary memory_hotplug_lock deadlock during initialization
Product DPDK
Version unspecified
Hardware All
OS All
Status UNCONFIRMED
Severity normal
Priority Normal
Component core
Assignee dev@dpdk.org
Reporter artemyko@nvidia.com
Target Milestone --- 

It seems the issue arose due to ch=
anges in the DPDK read-write lock
implementation. Following these changes, the RW-lock no longer supports
recursion, implying that a single thread shouldn't obtain a read lock if it
already possesses one. The problem arises during initialization: the
rte_eal_memory_init() function acquires the memory_hotplug_lock, and later =
on,
the sequence of calls eal_memalloc_init() -> rte_memseg_list_walk() acqu=
ires it
again without releasing it. This scenario introduces the risk of a potential
deadlock when concurrent write locks are applied to the same
memory_hotplug_lock. To address this locally, we resolved the issue by
replacing rte_memseg_list_walk() with rte_memseg_list_walk_thread_unsafe().

Reproduction:
Create mp_deadlock directory under dpdk/examples/. Then add main.c

/* SPDX-License-Identifier: BSD-3-Clause
* Copyright(c) 2010-2014 Intel Corporation
*/

#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <errno.h>
#include <sys/queue.h>

#include <rte_memory.h>
#include <rte_malloc.h>
#include <rte_launch.h>
#include <rte_eal.h>
#include <rte_per_lcore.h>
#include <rte_lcore.h>
#include <rte_debug.h>

/* Initialization of Environment Abstraction Layer (EAL). 8< */
int
main(int argc, char **argv)
{
        int ret;

        ret =3D rte_eal_init(argc, argv);
        if (ret < 0)
                rte_panic("Cannot init EAL\n");
        /* >8 End of initialization of Environment Abstraction Layer */

        if (rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY)
                getchar();
        else {
                if (rte_lcore_id() <=3D 1) {
                        int i =3D 0;
                        void *p;

                        while (1) {
                                p =3D rte_malloc_socket(NULL, 0x1000000, 0x=
1000,
-1);
                                rte_free(p);
                                printf("malloc %d times\n", i++);
                        }
                }
        }

        /* clean up the EAL */
        rte_eal_cleanup();

        return 0;
}

Compile: I followed https://doc.dpdk.org/guides/prog_guide/build_app.html and
some tips from related web page.

Run primary: ./examples/mp_deadlock/build/mp_deadlock -l 0 --file-prefix=3D=
dpdk1
--proc-type=3Dprimary
Run secondary 1: ./examples/mp_deadlock/build/mp_deadlock -l 1
--file-prefix=3Ddpdk1 --proc-type=3Dsecondary
Run secondary 2:=20
while true
do
        ./examples/mp_deadlock/build/mp_deadlock -l 2 --file-prefix=3Ddpdk1
--proc-type=3Dsecondary
done

Stack trace. It looks like the following caused deadlock.

#0  0x00007f850e97a3f2 in rte_mcfg_mem_write_lock () from
/usr/local/lib64/librte_eal.so.23
And
#0  0x00007f3f591b5362 in rte_mcfg_mem_read_lock () from
/usr/local/lib64/librte_eal.so.23

[root@fedora dpdk]# ps -ef | grep deadlock
root        7328    1004  0 20:47 pts/0    00:00:00 bash ./mp_deadlock1.sh
root        7329    7328  4 20:47 pts/0    00:00:01
./examples/mp_deadlock/build/mp_deadlock -l 0 --file-prefix=3Ddpdk1
--proc-type=3Dprimary
root        7333    5693  0 20:47 pts/4    00:00:00 bash ./mp_deadlock2.sh
root        7334    7333 94 20:47 pts/4    00:00:31
./examples/mp_deadlock/build/mp_deadlock -l 1 --file-prefix=3Ddpdk1
--proc-type=3Dsecondary
root        7337    5267  0 20:47 pts/1    00:00:00 bash ./mp_deadlock.sh
root        7338    7337 98 20:47 pts/1    00:00:29
./examples/mp_deadlock/build/mp_deadlock -l 2 --file-prefix=3Ddpdk1
--proc-type=3Dsecondary
root        7342    5480  0 20:47 pts/2    00:00:00 grep --color=3Dauto dea=
dlock
[root@fedora dpdk]# pstack 7329
Thread 4 (Thread 0x7f20ae487640 (LWP 7332) "telemetry-v2"):
#0  0x00007f20b200ae6f in accept () from /lib64/libc.so.6
#1  0x00007f20b1e004a3 in socket_listener () from
/usr/local/lib64/librte_telemetry.so.23
#2  0x00007f20b1f85b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f20b200a6a0 in clone3 () from /lib64/libc.so.6
Thread 3 (Thread 0x7f20aec88640 (LWP 7331) "rte_mp_handle"):
#0  0x00007f20b200b23d in recvmsg () from /lib64/libc.so.6
#1  0x00007f20b2137ecf in mp_handle () from /usr/local/lib64/librte_eal.so.=
23
#2  0x00007f20b1f85b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f20b200a6a0 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7f20af489640 (LWP 7330) "eal-intr-thread"):
#0  0x00007f20b2009c7e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f20b2141c54 in eal_intr_thread_main () from
/usr/local/lib64/librte_eal.so.23
#2  0x00007f20b1f85b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f20b200a6a0 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7f20b1df9900 (LWP 7329) "mp_deadlock"):
#0  0x00007f20b1ff984c in read () from /lib64/libc.so.6
#1  0x00007f20b1f7e914 in __GI__IO_file_underflow () from /lib64/libc.so.6
#2  0x00007f20b1f7f946 in _IO_default_uflow () from /lib64/libc.so.6
#3  0x00007f20b1f7a328 in getc () from /lib64/libc.so.6
#4  0x000000000040113e in main ()
[root@fedora dpdk]# pstack 7334
Thread 3 (Thread 0x7f850b4da640 (LWP 7336) "rte_mp_handle"):
#0  0x00007f850e85d23d in recvmsg () from /lib64/libc.so.6
#1  0x00007f850e989ecf in mp_handle () from /usr/local/lib64/librte_eal.so.=
23
#2  0x00007f850e7d7b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f850e85c6a0 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7f850bcdb640 (LWP 7335) "eal-intr-thread"):
#0  0x00007f850e85bc7e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f850e993c54 in eal_intr_thread_main () from
/usr/local/lib64/librte_eal.so.23
#2  0x00007f850e7d7b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f850e85c6a0 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7f850e64b900 (LWP 7334) "mp_deadlock"):
#0  0x00007f850e97a3f2 in rte_mcfg_mem_write_lock () from
/usr/local/lib64/librte_eal.so.23
#1  0x00007f850e984509 in malloc_heap_free () from
/usr/local/lib64/librte_eal.so.23
#2  0x00007f850e98508f in rte_free () from /usr/local/lib64/librte_eal.so.23
#3  0x0000000000401126 in main ()
[root@fedora dpdk]# pstack 7338
Thread 3 (Thread 0x7f3f55d15640 (LWP 7340) "rte_mp_handle"):
#0  0x00007f3f5909823d in recvmsg () from /lib64/libc.so.6
#1  0x00007f3f591c4ecf in mp_handle () from /usr/local/lib64/librte_eal.so.=
23
#2  0x00007f3f59012b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f3f590976a0 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3f56516640 (LWP 7339) "eal-intr-thread"):
#0  0x00007f3f59096c7e in epoll_wait () from /lib64/libc.so.6
#1  0x00007f3f591cec54 in eal_intr_thread_main () from
/usr/local/lib64/librte_eal.so.23
#2  0x00007f3f59012b17 in start_thread () from /lib64/libc.so.6
#3  0x00007f3f590976a0 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3f58e86900 (LWP 7338) "mp_deadlock"):
#0  0x00007f3f591b5362 in rte_mcfg_mem_read_lock () from
/usr/local/lib64/librte_eal.so.23
#1  0x00007f3f591b6bf2 in rte_memseg_list_walk () from
/usr/local/lib64/librte_eal.so.23
#2  0x00007f3f591d2f65 in eal_memalloc_init () from
/usr/local/lib64/librte_eal.so.23
#3  0x00007f3f591b741b in rte_eal_memory_init () from
/usr/local/lib64/librte_eal.so.23
#4  0x00007f3f591aab64 in rte_eal_init.cold () from
/usr/local/lib64/librte_eal.so.23
#5  0x00000000004010d9 in main ()

You are receiving this mail because:
  • You are the assignee for the bug.
=20=20=20=20=20=20=20=20=20=20
= --16927993430.bCf0221a5.1434830--