From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 49738A04FD; Sat, 30 Jul 2022 18:17:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F3E9B4021E; Sat, 30 Jul 2022 18:17:38 +0200 (CEST) Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by mails.dpdk.org (Postfix) with ESMTP id 896044014F for ; Sat, 30 Jul 2022 18:17:37 +0200 (CEST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 5D68F207CF; Sat, 30 Jul 2022 16:17:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1659197857; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mNw9vFtos6oui4fkVTFgoyuuZMsoRfxGpA0p3gGDmX4=; b=10jVqxGjrvria174MIg96I6XNWCDxmqfAqU0gPC5jmj1miD1UB1I5lEFSY62Mml+F4FhI0 ZXJfTTDWiYkS6Kz2XW5cOx95eyDh22TrtmX7ZLlzVUywZSGXCe0AZ4Kdma09fDYR4MPA66 r3kDK5FORRD56esZsJeltYbPtpLMUBo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1659197857; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mNw9vFtos6oui4fkVTFgoyuuZMsoRfxGpA0p3gGDmX4=; b=sK1/CjjCRgAPFIhifcMchDfnewbNzCzKzGKZo7a5E0yBw0wgjy2P96Utk355awvHeWwUoB kXv1nhlnwMIVMMBg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 3BC511331F; Sat, 30 Jul 2022 16:17:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id o4vjDKFZ5WKFHAAAMHmgww (envelope-from ); Sat, 30 Jul 2022 16:17:37 +0000 Message-ID: Date: Sat, 30 Jul 2022 18:17:36 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 To: dev@dpdk.org Content-Language: en-US From: Claudio Fontana Subject: segfault in ovs in setup with DPDK, qemu vhost-user Cc: Marco Varlese Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hello all, with the latest DPDK, openvswitch and qemu DPDK tag v22.07 openvswitch tag v2.17.1 qemu v7.1-git 22.07.2022 and a DPDK setup which involves also an ubuntu guest with DPDK 16.11 test-pmd application (but also verified with DPDK 19.x), with an external traffic generator to cause some load, I am able to cause a segfault in OVS (ovs-vswitchd) inside the DPDK libraries by doing (from the guest): bind the device, start testpmd, SIGKILL of testpmd, immediately restart testpmd, rinse and repeat. Once every few restarts, the following segfault happens (may take anything from a few seconds to minutes): Thread 153 "pmd-c88/id:150" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f64e5e6b700 (LWP 141373)] rte_mov128blocks (n=2048, src=0xc , dst=0x150da4480 "h\005\312❇\377\377\377\377\377\377\b") at ../lib/eal/x86/include/rte_memcpy.h:384 384 ../lib/eal/x86/include/rte_memcpy.h: No such file or directory. (gdb) bt #0 rte_mov128blocks (n=2048, src=0xc , dst=0x150da4480 "h\005\312❇\377\377\377\377\377\377\b") at ../lib/eal/x86/include/rte_memcpy.h:384 #1 rte_memcpy_generic (n=2048, src=0xc, dst=0x150da4480) at ../lib/eal/x86/include/rte_memcpy.h:484 #2 rte_memcpy (n=2048, src=0xc, dst=) at ../lib/eal/x86/include/rte_memcpy.h:851 #3 sync_fill_seg (to_desc=false, cpy_len=2048, buf_iova=, buf_addr=12, mbuf_offset=0, m=0x150da4140, vq=0x2200400680, dev=0x2200d3d740) at ../lib/vhost/virtio_net.c:1119 #4 desc_to_mbuf (is_async=false, slot_idx=0, legacy_ol_flags=true, mbuf_pool=0x17fe7df00, m=0x150da4140, nr_vec=, buf_vec=0x7f64e5e67ca0, vq=0x2200400680, dev=0x2200d3d740) at ../lib/vhost/virtio_net.c:2747 #5 virtio_dev_tx_split (legacy_ol_flags=true, count=, count@entry=0, pkts=pkts@entry=0x0, mbuf_pool=mbuf_pool@entry=0x150da4140, vq=vq@entry=0xe5e67d34, dev=dev@entry=0x7f64e5e694d0) at ../lib/vhost/virtio_net.c:2943 #6 virtio_dev_tx_split_legacy (dev=dev@entry=0x2200d3d740, vq=vq@entry=0x2200400680, mbuf_pool=mbuf_pool@entry=0x17fe7df00, pkts=pkts@entry=0x7f64e5e69600, count=count@entry=32) at ../lib/vhost/virtio_net.c:2979 #7 0x00007f676fea0fef in rte_vhost_dequeue_burst (vid=vid@entry=0, queue_id=queue_id@entry=1, mbuf_pool=0x17fe7df00, pkts=pkts@entry=0x7f64e5e69600, count=count@entry=32) at ../lib/vhost/virtio_net.c:3331 #8 0x00007f6772005a62 in netdev_dpdk_vhost_rxq_recv (rxq=, batch=0x7f64e5e695f0, qfill=0x0) at ../lib/netdev-dpdk.c:2393 #9 0x00007f6771f38116 in netdev_rxq_recv (rx=, batch=batch@entry=0x7f64e5e695f0, qfill=) at ../lib/netdev.c:727 #10 0x00007f6771f03d96 in dp_netdev_process_rxq_port (pmd=pmd@entry=0x7f64e5e6c010, rxq=0x254d730, port_no=2) at ../lib/dpif-netdev.c:5317 #11 0x00007f6771f04239 in pmd_thread_main (f_=) at ../lib/dpif-netdev.c:6945 #12 0x00007f6771f92aff in ovsthread_wrapper (aux_=) at ../lib/ovs-thread.c:422 #13 0x00007f6771c1b6ea in start_thread () from /lib64/libpthread.so.0 #14 0x00007f6771933a8f in clone () from /lib64/libc.so.6 When run in gdb as shown above, ovs-vswitchd on the host gets a SIGSEGV and drops to gdb as shown above, so as a result QEMU stops when trying to read a response from ovs as such: 0 0x00007f0a093991e9 in poll () from target:/lib64/libc.so.6 #1 0x00007f0a0b06c9a9 in ?? () from target:/usr/lib64/libglib-2.0.so.0 #2 0x00007f0a0b06ccf2 in g_main_loop_run () from target:/usr/lib64/libglib-2.0.so.0 #3 0x0000561a5cd04747 in vhost_user_read (dev=dev@entry=0x561a5e640df0, msg=msg@entry=0x7f09ff7fd160) at ../hw/virtio/vhost-user.c:406 #4 0x0000561a5cd04c7e in vhost_user_get_vring_base (dev=0x561a5e640df0, ring=0x7f09ff7fd428) at ../hw/virtio/vhost-user.c:1261 #5 0x0000561a5cd0043f in vhost_virtqueue_stop (dev=dev@entry=0x561a5e640df0, vdev=vdev@entry=0x561a5f78ae50, vq=0x561a5e641070, idx=0) at ../hw/virtio/vhost.c:1216 #6 0x0000561a5cd034fa in vhost_dev_stop (hdev=hdev@entry=0x561a5e640df0, vdev=vdev@entry=0x561a5f78ae50) at ../hw/virtio/vhost.c:1872 #7 0x0000561a5cb623fa in vhost_net_stop_one (net=0x561a5e640df0, dev=dev@entry=0x561a5f78ae50) at ../hw/net/vhost_net.c:315 #8 0x0000561a5cb6295e in vhost_net_stop (dev=dev@entry=0x561a5f78ae50, ncs=0x561a5f808970, data_queue_pairs=data_queue_pairs@entry=4, cvq=cvq@entry=0) at ../hw/net/vhost_net.c:427 #9 0x0000561a5cccef79 in virtio_net_vhost_status (status=, n=0x561a5f78ae50) at ../hw/net/virtio-net.c:298 #10 virtio_net_set_status (vdev=0x561a5f78ae50, status=0 '\000') at ../hw/net/virtio-net.c:372 #11 0x0000561a5ccfb36b in virtio_set_status (vdev=vdev@entry=0x561a5f78ae50, val=val@entry=0 '\000') at ../hw/virtio/virtio.c:1997 #12 0x0000561a5cbfff29 in virtio_pci_common_write (opaque=0x561a5f782a90, addr=, val=0, size=) at ../hw/virtio/virtio-pci.c:1294 #13 0x0000561a5cd25fbf in memory_region_write_accessor (mr=0x561a5f7835c0, addr=20, value=, size=1, shift=, mask=, attrs=...) at ../softmmu/memory.c:492 #14 0x0000561a5cd22950 in access_with_adjusted_size (addr=addr@entry=20, value=value@entry=0x7f09ff7fd6f8, size=size@entry=1, access_size_min=, access_size_max=, access_fn=access_fn@entry=0x561a5cd25f6d , mr=0x561a5f7835c0, attrs=...) Some additional info about the setup: Host is running SUSE Linux Enterprise Server 15SP3, with DPDK, openvswitch and QEMU packages replaced with latest upstream releases. Guest is running the following libvirt VM: ubuntu20.04-3 971953a5-bd24-4856-a117-87c791a09580 4194304 4194304 5 /machine hvm IvyBridge-IBRS Intel destroy restart destroy /usr/bin/qemu-system-x86_64