From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7FC0EA034C; Mon, 1 Aug 2022 13:57:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 347BF41101; Mon, 1 Aug 2022 13:57:47 +0200 (CEST) Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by mails.dpdk.org (Postfix) with ESMTP id C38B64067B for ; Mon, 1 Aug 2022 13:57:45 +0200 (CEST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9B4672075D; Mon, 1 Aug 2022 11:57:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1659355065; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c+ph+oMgytaTLUWEXwlJ+aTEqhxrpCrk97EDZidOTG8=; b=qRgDt4Eg3IWyD0v1dEHm9kpU8ga+rKArTdodnXgo88twKw322aDM5sx7gArIyShrAB4GZq goIZ5eZ2U5nBhXArXmApZGYqmvMrkl72pp2eEneJ60aNRIGd9zpVLKCk6I2iXtETSU22sw LaTaikv7TpjBb3OmIGWu5siwjsChYM8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1659355065; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c+ph+oMgytaTLUWEXwlJ+aTEqhxrpCrk97EDZidOTG8=; b=+9u+f03b2H4jEKz/tq3BGr2Wii8rXihdaK4wWEX+NDtETeyJV6PreX7DEogfEtdme7hMyW Fd8m5AQ44IZaopBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 77E6B13AAE; Mon, 1 Aug 2022 11:57:45 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id fgGXG7m/52IoEAAAMHmgww (envelope-from ); Mon, 01 Aug 2022 11:57:45 +0000 Message-ID: <14f450ba-2727-1afc-8de8-908c0b19078d@suse.de> Date: Mon, 1 Aug 2022 13:57:44 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.4.0 Subject: Re: segfault in ovs in setup with DPDK, qemu vhost-user Content-Language: en-US From: Claudio Fontana To: dev@dpdk.org Cc: Marco Varlese References: <7400eb9e-e677-3228-5ff2-f4b7ac05b2ba@suse.de> In-Reply-To: <7400eb9e-e677-3228-5ff2-f4b7ac05b2ba@suse.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 7/30/22 18:23, Claudio Fontana wrote: > On 7/30/22 18:17, Claudio Fontana wrote: >> Hello all, >> >> with the latest DPDK, openvswitch and qemu >> >> DPDK tag v22.07 >> openvswitch tag v2.17.1 >> qemu v7.1-git 22.07.2022 >> >> and a DPDK setup which involves also an ubuntu guest with DPDK 16.11 test-pmd application (but also verified with DPDK 19.x), >> with an external traffic generator to cause some load, >> >> I am able to cause a segfault in OVS (ovs-vswitchd) inside the DPDK libraries by doing (from the guest): >> >> bind the device, start testpmd, >> SIGKILL of testpmd, >> immediately restart testpmd, >> rinse and repeat. >> >> Once every few restarts, the following segfault happens (may take anything from a few seconds to minutes): >> >> >> Thread 153 "pmd-c88/id:150" received signal SIGSEGV, Segmentation fault. >> [Switching to Thread 0x7f64e5e6b700 (LWP 141373)] >> rte_mov128blocks (n=2048, src=0xc , dst=0x150da4480 "h\005\312❇\377\377\377\377\377\377\b") at ../lib/eal/x86/include/rte_memcpy.h:384 >> 384 ../lib/eal/x86/include/rte_memcpy.h: No such file or directory. >> (gdb) bt >> #0 rte_mov128blocks (n=2048, src=0xc , >> dst=0x150da4480 "h\005\312❇\377\377\377\377\377\377\b") at ../lib/eal/x86/include/rte_memcpy.h:384 >> #1 rte_memcpy_generic (n=2048, src=0xc, dst=0x150da4480) at ../lib/eal/x86/include/rte_memcpy.h:484 >> #2 rte_memcpy (n=2048, src=0xc, dst=) at ../lib/eal/x86/include/rte_memcpy.h:851 >> #3 sync_fill_seg (to_desc=false, cpy_len=2048, buf_iova=, buf_addr=12, mbuf_offset=0, m=0x150da4140, >> vq=0x2200400680, dev=0x2200d3d740) at ../lib/vhost/virtio_net.c:1119 >> #4 desc_to_mbuf (is_async=false, slot_idx=0, legacy_ol_flags=true, mbuf_pool=0x17fe7df00, m=0x150da4140, nr_vec=, >> buf_vec=0x7f64e5e67ca0, vq=0x2200400680, dev=0x2200d3d740) at ../lib/vhost/virtio_net.c:2747 >> #5 virtio_dev_tx_split (legacy_ol_flags=true, count=, count@entry=0, pkts=pkts@entry=0x0, >> mbuf_pool=mbuf_pool@entry=0x150da4140, vq=vq@entry=0xe5e67d34, dev=dev@entry=0x7f64e5e694d0) at ../lib/vhost/virtio_net.c:2943 >> #6 virtio_dev_tx_split_legacy (dev=dev@entry=0x2200d3d740, vq=vq@entry=0x2200400680, mbuf_pool=mbuf_pool@entry=0x17fe7df00, >> pkts=pkts@entry=0x7f64e5e69600, count=count@entry=32) at ../lib/vhost/virtio_net.c:2979 >> #7 0x00007f676fea0fef in rte_vhost_dequeue_burst (vid=vid@entry=0, queue_id=queue_id@entry=1, mbuf_pool=0x17fe7df00, >> pkts=pkts@entry=0x7f64e5e69600, count=count@entry=32) at ../lib/vhost/virtio_net.c:3331 >> #8 0x00007f6772005a62 in netdev_dpdk_vhost_rxq_recv (rxq=, batch=0x7f64e5e695f0, qfill=0x0) >> at ../lib/netdev-dpdk.c:2393 >> #9 0x00007f6771f38116 in netdev_rxq_recv (rx=, batch=batch@entry=0x7f64e5e695f0, qfill=) >> at ../lib/netdev.c:727 >> #10 0x00007f6771f03d96 in dp_netdev_process_rxq_port (pmd=pmd@entry=0x7f64e5e6c010, rxq=0x254d730, port_no=2) >> at ../lib/dpif-netdev.c:5317 >> #11 0x00007f6771f04239 in pmd_thread_main (f_=) at ../lib/dpif-netdev.c:6945 >> #12 0x00007f6771f92aff in ovsthread_wrapper (aux_=) at ../lib/ovs-thread.c:422 >> #13 0x00007f6771c1b6ea in start_thread () from /lib64/libpthread.so.0 >> #14 0x00007f6771933a8f in clone () from /lib64/libc.so.6 >> >> When run in gdb as shown above, ovs-vswitchd on the host gets a SIGSEGV and drops to gdb as shown above, >> so as a result QEMU stops when trying to read a response from ovs as such: >> >> 0 0x00007f0a093991e9 in poll () from target:/lib64/libc.so.6 >> #1 0x00007f0a0b06c9a9 in ?? () from target:/usr/lib64/libglib-2.0.so.0 >> #2 0x00007f0a0b06ccf2 in g_main_loop_run () from target:/usr/lib64/libglib-2.0.so.0 >> #3 0x0000561a5cd04747 in vhost_user_read (dev=dev@entry=0x561a5e640df0, msg=msg@entry=0x7f09ff7fd160) >> at ../hw/virtio/vhost-user.c:406 >> #4 0x0000561a5cd04c7e in vhost_user_get_vring_base (dev=0x561a5e640df0, ring=0x7f09ff7fd428) >> at ../hw/virtio/vhost-user.c:1261 >> #5 0x0000561a5cd0043f in vhost_virtqueue_stop (dev=dev@entry=0x561a5e640df0, vdev=vdev@entry=0x561a5f78ae50, >> vq=0x561a5e641070, idx=0) at ../hw/virtio/vhost.c:1216 >> #6 0x0000561a5cd034fa in vhost_dev_stop (hdev=hdev@entry=0x561a5e640df0, vdev=vdev@entry=0x561a5f78ae50) >> at ../hw/virtio/vhost.c:1872 >> #7 0x0000561a5cb623fa in vhost_net_stop_one (net=0x561a5e640df0, dev=dev@entry=0x561a5f78ae50) >> at ../hw/net/vhost_net.c:315 >> #8 0x0000561a5cb6295e in vhost_net_stop (dev=dev@entry=0x561a5f78ae50, ncs=0x561a5f808970, >> data_queue_pairs=data_queue_pairs@entry=4, cvq=cvq@entry=0) at ../hw/net/vhost_net.c:427 >> #9 0x0000561a5cccef79 in virtio_net_vhost_status (status=, n=0x561a5f78ae50) >> at ../hw/net/virtio-net.c:298 >> #10 virtio_net_set_status (vdev=0x561a5f78ae50, status=0 '\000') at ../hw/net/virtio-net.c:372 >> #11 0x0000561a5ccfb36b in virtio_set_status (vdev=vdev@entry=0x561a5f78ae50, val=val@entry=0 '\000') >> at ../hw/virtio/virtio.c:1997 >> #12 0x0000561a5cbfff29 in virtio_pci_common_write (opaque=0x561a5f782a90, addr=, val=0, >> size=) at ../hw/virtio/virtio-pci.c:1294 >> #13 0x0000561a5cd25fbf in memory_region_write_accessor (mr=0x561a5f7835c0, addr=20, value=, size=1, >> shift=, mask=, attrs=...) at ../softmmu/memory.c:492 >> #14 0x0000561a5cd22950 in access_with_adjusted_size (addr=addr@entry=20, value=value@entry=0x7f09ff7fd6f8, >> size=size@entry=1, access_size_min=, access_size_max=, >> access_fn=access_fn@entry=0x561a5cd25f6d >> , mr=0x561a5f7835c0, attrs=...) >> >> Some additional info about the setup: >> >> Host is running SUSE Linux Enterprise Server 15SP3, with DPDK, openvswitch and QEMU packages replaced with latest upstream releases. >> >> Guest is running the following libvirt VM: >> >> >> ubuntu20.04-3 >> 971953a5-bd24-4856-a117-87c791a09580 >> >> >> >> >> >> 4194304 >> 4194304 >> >> >> >> >> >> 5 >> >> /machine >> >> >> hvm >> >> >> >> >> >> >> >> >> IvyBridge-IBRS >> Intel >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> destroy >> restart >> destroy >> >> >> >> >> >> /usr/bin/qemu-system-x86_64 >> >> >> >> >> >> >>
>> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >> >> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >>
>> >> >> >> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >>
>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>
>> >> >> >>
>> >> >> >> >> >> >> >> >> >> >> >> >>
>> >>