From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2F286A0A0C; Fri, 14 May 2021 10:17:16 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 14CD640042; Fri, 14 May 2021 10:17:16 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id A7AB840041 for ; Fri, 14 May 2021 10:17:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1620980234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=J8CLgYrkCBBsm551e9tJvqmdfEuXWnBV+KA2MgigPXo=; b=S89buYM8+iIgK6SFYlpgMR9zAPSuojf/SSfolFVgQvRIig1V5FQiML1qiFiRCVjrmwuWLa z5TiMYMKqXyi0s0NxmpDXSyTDjrtIAvnBiZexy9opNBVpC7bgCj+qezrJoX/iT9YzRV1x+ 0qH2CMDHgSdWW44iUN1fCJmbcoNm8Wc= Received: from mail-ua1-f72.google.com (mail-ua1-f72.google.com [209.85.222.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-448-mbnRODjZOc21HyzfmdHy8g-1; Fri, 14 May 2021 04:17:12 -0400 X-MC-Unique: mbnRODjZOc21HyzfmdHy8g-1 Received: by mail-ua1-f72.google.com with SMTP id v7-20020a9f31870000b029020b2d4ea5d9so3504842uad.8 for ; Fri, 14 May 2021 01:17:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=J8CLgYrkCBBsm551e9tJvqmdfEuXWnBV+KA2MgigPXo=; b=G7C37NL8W1CsHe/eZklJVkIAJ2dJ7vqGb6E7LNgbu4XAm6V0hTRm2UZ5IDs7qHV6Hv yAj0DJdM5oET4KK7RbojXrJ7Zz1BdQjdvkEtqu57tF/0oS1ieA8MLbK2Ufdg/6F4afRg AzTS4pS0zxNlQ2WpGGy/JgUMaNdJoSjLdlG3xSCs4n76zv1a+kahAXZhscjWaKSMw1Xp MedrkZrkvfVNH1OReswZFMyP2isNZqEkvwFjTV1vognuZ22o/MpGwYyislHDh7uufK6f RGTwXUH7nhtNras1aF3rQxsfh6ZIg9SqBQ9QHRHGyqqbMqCsgmVY23V3BYtJdhp2aAEJ zZgA== X-Gm-Message-State: AOAM531LG8/U8mvxzYbKpNJjzCEov7D95iLKAH5azX6K/su4ZbnZoFWF qNWezUHNiYsOp7jpOV7XXF+gu1oWSZk+h0TqPasfZ+EwN8/LU1vNhZaWEYOYqVwCgzoiUhR7LFN EJzLS2U+lGkZzJgPhOJg= X-Received: by 2002:a67:fb53:: with SMTP id e19mr42534431vsr.10.1620980231602; Fri, 14 May 2021 01:17:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwFmbEMQWoAimUE91FpRCbwBHAWt0lh+RCbghpgDziAMjNvBqFqo/1uQTFYMRrkmeEQ8xluqpJWAlp0T1tpACg= X-Received: by 2002:a67:fb53:: with SMTP id e19mr42534385vsr.10.1620980231196; Fri, 14 May 2021 01:17:11 -0700 (PDT) MIME-Version: 1.0 References: <20210513122826.49910-1-chenbo.xia@intel.com> In-Reply-To: From: David Marchand Date: Fri, 14 May 2021 10:16:59 +0200 Message-ID: To: Chenbo Xia , Maxime Coquelin Cc: dev , Kevin Traynor , Pei Zhang , "Yigit, Ferruh" , Thomas Monjalon Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dmarchan@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] vhost: fix wrong IOTLB initialization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, May 13, 2021 at 4:11 PM David Marchand wrote: > On Thu, May 13, 2021 at 2:38 PM Chenbo Xia wrote: > > > > This patch fixes an issue of application crash because of vhost iotlb > > not initialized when virtio has multiqueue enabled. > > > > iotlb messages will be sent when some queues are not enabled. If we > > initialize iotlb in vhost_user_set_vring_num, it could happen that > > iotlb update comes when iotlb pool of disabled queues are not > > initialized. > > This makes the problem I reproduced disappear at init, but I noticed > the segfault after restarting testpmd once. > And a little bit after this, my vm crashed. > > This is not systematic, so I guess there is some condition with how > the virtio device is initialised in the vm. The crash is systematic (not sure what I missed yesterday, but I always get it with simple steps below). Full logs: # dpdk-testpmd --vdev net_vhost0,iface=/var/lib/vhost_sockets/vhost0,client=1,iommu-support=1,queues=2 -w 0:0:0.0 --log-level=lib.vhost.config:debug -- -ia --rxq=2 VHOST_CONFIG: vhost-user client: socket created, fd: 31 VHOST_CONFIG: failed to connect to /var/lib/vhost_sockets/vhost0: No such file or directory VHOST_CONFIG: /var/lib/vhost_sockets/vhost0: reconnecting... # start vm (with virtio device in the guest OS bound to kernel kmod, i.e. no special configuration) # testpmd logs: testpmd> VHOST_CONFIG: /var/lib/vhost_sockets/vhost0: connected VHOST_CONFIG: new device, handle is 0 VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM VHOST_CONFIG: Processing VHOST_USER_GET_QUEUE_NUM succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded. VHOST_CONFIG: read message VHOST_USER_SET_OWNER VHOST_CONFIG: Processing VHOST_USER_SET_OWNER succeeded. VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:36 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:37 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded. VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:2 file:35 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:3 file:39 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 0 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 1 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 0 to qp idx: 2 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 0 to qp idx: 3 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 0 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 1 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 0 to qp idx: 2 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 0 to qp idx: 3 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_FEATURES VHOST_CONFIG: negotiated Virtio features: 0x370607f83 VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_0 VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_1 VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_2 VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_3 VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE VHOST_CONFIG: guest memory region size: 0x80000000 guest physical addr: 0x0 guest virtual addr: 0x7f2400000000 host virtual addr: 0x7fff40000000 mmap addr : 0x7fff40000000 mmap size : 0x80000000 mmap align: 0x40000000 mmap off : 0x0 VHOST_CONFIG: guest memory region size: 0x180000000 guest physical addr: 0x100000000 guest virtual addr: 0x7f2480000000 host virtual addr: 0x7ffdc0000000 mmap addr : 0x7ffd40000000 mmap size : 0x200000000 mmap align: 0x40000000 mmap off : 0x80000000 VHOST_CONFIG: Processing VHOST_USER_SET_MEM_TABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:0 file:42 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:43 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:1 file:36 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:44 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 0 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 1 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_FEATURES VHOST_CONFIG: negotiated Virtio features: 0x370607f83 VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_0 VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_1 VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_2 VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_3 VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:2 file:37 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:2 file:45 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:3 file:35 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:3 file:46 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map avail ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map avail ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) mapped address desc: 0x7fff1d886000 VHOST_CONFIG: (0) mapped address avail: 0x7fff1d887000 VHOST_CONFIG: (0) mapped address used: 0x7fff1d887240 VHOST_CONFIG: (0) log_guest_addr: 0 VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) mapped address desc: 0x7fff1d89e000 VHOST_CONFIG: (0) mapped address avail: 0x7fff1d89f000 VHOST_CONFIG: (0) mapped address used: 0x7fff1d89f240 VHOST_CONFIG: (0) log_guest_addr: 0 VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) mapped address desc: 0x7fff1d8da000 VHOST_CONFIG: (0) mapped address avail: 0x7fff1d8db000 VHOST_CONFIG: (0) mapped address used: 0x7fff1d8db240 VHOST_CONFIG: (0) log_guest_addr: 0 VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. Port 0: queue state event VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) mapped address desc: 0x7fff1e198000 VHOST_CONFIG: (0) mapped address avail: 0x7fff1e199000 VHOST_CONFIG: (0) mapped address used: 0x7fff1e199240 VHOST_CONFIG: (0) log_guest_addr: 0 VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. Port 0: queue state event VHOST_CONFIG: virtio is now ready for processing. Port 0: link state change event VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 0 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 1 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 2 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. Port 0: queue state event VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 3 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. Port 0: queue state event VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. So far, everything looks good. Now I quit testpmd. VHOST_CONFIG: free connfd = 31 for device '/var/lib/vhost_sockets/vhost0' And I restart testpmd with the same command as above: VHOST_CONFIG: vhost-user client: socket created, fd: 31 VHOST_CONFIG: new device, handle is 0 Port 0: 56:48:4F:53:54:00 Checking link statuses... VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM VHOST_CONFIG: Processing VHOST_USER_GET_QUEUE_NUM succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded. VHOST_CONFIG: read message VHOST_USER_SET_OWNER VHOST_CONFIG: Processing VHOST_USER_SET_OWNER succeeded. VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:36 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:37 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded. VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:2 file:35 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:3 file:39 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_FEATURES VHOST_CONFIG: negotiated Virtio features: 0x370607f83 VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_0 VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_1 VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_2 VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_3 VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE VHOST_CONFIG: guest memory region size: 0x80000000 guest physical addr: 0x0 guest virtual addr: 0x7f2400000000 host virtual addr: 0x7fff40000000 mmap addr : 0x7fff40000000 mmap size : 0x80000000 mmap align: 0x40000000 mmap off : 0x0 VHOST_CONFIG: guest memory region size: 0x180000000 guest physical addr: 0x100000000 guest virtual addr: 0x7f2480000000 host virtual addr: 0x7ffdc0000000 mmap addr : 0x7ffd40000000 mmap size : 0x200000000 mmap align: 0x40000000 mmap off : 0x80000000 VHOST_CONFIG: Processing VHOST_USER_SET_MEM_TABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:0 file:42 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:43 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:1 file:36 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:44 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 0 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE VHOST_CONFIG: set queue enable: 1 to qp idx: 1 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_FEATURES VHOST_CONFIG: negotiated Virtio features: 0x40000000 VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 off VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:2 file:35 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:2 file:37 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK VHOST_CONFIG: vring kick idx:3 file:39 VHOST_CONFIG: (0) failed to map desc ring. VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded. VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:3 file:45 VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded. VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG Thread 9 "vhost-events" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffee7fc400 (LWP 101339)] 0x0000000000e30803 in rte_mempool_default_cache (lcore_id=4294967295, mp=0x0) at ../lib/mempool/rte_mempool.h:1244 1244 if (mp->cache_size == 0) Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8_3.2.x86_64 libibverbs-29.0-3.el8.x86_64 libnl3-3.5.0-1.el8.x86_64 numactl-libs-2.0.12-11.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64 (gdb) bt full #0 0x0000000000e30803 in rte_mempool_default_cache (lcore_id=4294967295, mp=0x0) at ../lib/mempool/rte_mempool.h:1244 No locals. #1 rte_mempool_get_bulk (n=1, obj_table=0x7fffee7f5e98, mp=0x0) at ../lib/mempool/rte_mempool.h:1533 cache = 0x0 cache = #2 rte_mempool_get (obj_p=0x7fffee7f5e98, mp=0x0) at ../lib/mempool/rte_mempool.h:1561 No locals. #3 vhost_user_iotlb_cache_insert (vq=0x1673bff00, iova=10159157248, uaddr=140733688868864, size=4096, perm=3 '\003') at ../lib/vhost/iotlb.c:164 node = 0x1673c08c0 new_node = 0x1673671f0 ret = 0 #4 0x000000000100d51d in vhost_user_iotlb_msg (pdev=0x7fffee7f92b8, msg=0x7fffee7f9010, main_fd=31) at ../lib/vhost/vhost_user.c:2409 vq = 0x1673bff00 dev = 0x1673c08c0 imsg = 0x7fffee7f901c i = 2 vva = 140733688868864 len = 4096 #5 0x000000000100e31e in vhost_user_msg_handler (vid=0, fd=31) at ../lib/vhost/vhost_user.c:2882 dev = 0x1673c08c0 msg = {request = {master = 22, slave = 22}, flags = 9, size = 32, payload = {u64 = 10159157248, state = {index = 1569222656, num = 2}, addr = {index = 1569222656, flags = 2, desc_user_addr = 4096, used_user_addr = 139800607223808, avail_user_addr = 7907112783476097539, log_guest_addr = 10159256128}, memory = {nregions = 1569222656, padding = 2, regions = {{guest_phys_addr = 4096, memory_size = 139800607223808, userspace_addr = 7907112783476097539, mmap_offset = 10159256128}, {guest_phys_addr = 4294967296, memory_size = 6442450944, userspace_addr = 139794743033856, mmap_offset = 2147483648}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 4294967296}, { guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 4294967296, userspace_addr = 0, mmap_offset = 3419188017980506112}}}, log = {mmap_size = 10159157248, mmap_offset = 4096}, iotlb = {iova = 10159157248, size = 4096, uaddr = 139800607223808, perm = 3 '\003', type = 2 '\002'}, crypto_session = {session_id = 10159157248, op_code = 4096, cipher_algo = 0, cipher_key_len = 3716706304, hash_algo = 32549, digest_len = 3523936771, auth_key_len = 1841018158, aad_len = 1569321536, op_type = 2 '\002', dir = 0 '\000', hash_mode = 0 '\000', chaining_dir = 0 '\000', ciphe_key = 0x100000000 "\356o.\001\003", auth_key = 0x180000000 "", cipher_key_buf = "\000\000\000\200$\177\000\000\000\000\000\200", '\000' , auth_key_buf = '\000' , "\001", '\000' , "\001", '\000' , "ces/system/node", '\000' ...}, area = {u64 = 10159157248, size = 4096, offset = 139800607223808}, inflight = {mmap_size = 10159157248, mmap_offset = 4096, num_queues = 28672, queue_size = 56712}}, fds = {-1, -1, -1, -1, -1, -1, -1, -1}, fd_num = 0} vdpa_dev = 0x0 ret = 0 unlock_required = 0 handled = false request = 22 i = 4 #6 0x0000000000e41f75 in vhost_user_read_cb (connfd=31, dat=0x6e28510, remove=0x7fffee7f9394) at ../lib/vhost/socket.c:309 conn = 0x6e28510 vsocket = 0x6e1f140 ret = 1 #7 0x0000000000e0f9c4 in fdset_event_dispatch (arg=0x66f9a60 ) at ../lib/vhost/fd_man.c:286 i = 1 pfd = 0x66f9a68 pfdentry = 0x66fba88 rcb = 0xe41f3c wcb = 0x0 dat = 0x6e28510 fd = 31 numfds = 2 ... And qemu crashes right after this: (gdb) bt full #0 0x000056269a497160 in vhost_device_iotlb_miss (dev=dev@entry=0x56269d6ed800, iova=10159497216, write=) at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost.c:944 iotlb = uaddr = len = ret = -14 _rcu_read_auto = 0x1 #1 0x000056269a499361 in vhost_backend_handle_iotlb_msg (imsg=0x7ffd5eddeea0, dev=0x56269d6ed800) at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-backend.c:351 ret = ret = #2 vhost_backend_handle_iotlb_msg (dev=dev@entry=0x56269d6ed800, imsg=imsg@entry=0x7ffd5eddeea0) at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-backend.c:344 ret = 0 #3 0x000056269a499f0b in slave_read (opaque=0x56269d6ed800) at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-user.c:1048 dev = 0x56269d6ed800 u = 0x56269cd3f7b0 hdr = {request = VHOST_USER_GET_FEATURES, flags = 1, size = 32} payload = {u64 = 10159497216, state = {index = 1569562624, num = 2}, addr = {index = 1569562624, flags = 2, desc_user_addr = 0, used_user_addr = 0, avail_user_addr = 259, log_guest_addr = 0}, memory = { nregions = 1569562624, padding = 2, regions = {{guest_phys_addr = 0, memory_size = 0, userspace_addr = 259, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}}}, log = {mmap_size = 10159497216, mmap_offset = 0}, iotlb = {iova = 10159497216, size = 0, uaddr = 0, perm = 3 '\003', type = 1 '\001'}, config = {offset = 1569562624, size = 2, flags = 0, region = '\000' , "\003\001", '\000' }, session = { session_id = 10159497216, session_setup_data = {op_code = 0, cipher_alg = 0, key_len = 0, hash_alg = 0, hash_result_len = 259, auth_key_len = 0, add_len = 0, op_type = 0 '\000', direction = 0 '\000', hash_mode = 0 '\000', alg_chain_order = 0 '\000', cipher_key = 0x0, auth_key = 0x0}, key = '\000' , auth_key = '\000' }, area = { u64 = 10159497216, size = 0, offset = 0}, inflight = {mmap_size = 10159497216, mmap_offset = 0, num_queues = 0, queue_size = 0}} size = ret = 0 iov = {iov_base = 0x7ffd5eddee04, iov_len = 12} msgh = {msg_name = 0x0, msg_namelen = 0, msg_iov = 0x7ffd5eddee10, msg_iovlen = 1, msg_control = 0x7ffd5eddf120, msg_controllen = 0, msg_flags = 0} fd = {-1, -1, -1, -1, -1, -1, -1, -1} control = "^\a\022\000\000\000\000\000\226\070\270\t\000\000\000\000\334\345\061\234&V\000\000\000x\v\322.\265\273mP\347\061\234&V\000\000\000\000\000\000\000\000\000" cmsg = i = fdsize = 0 ... -- David Marchand