From: Marco Varlese <marco.varlese@suse.com>
To: "Xia, Chenbo" <chenbo.xia@intel.com>,
"ktraynor@redhat.com" <ktraynor@redhat.com>,
"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
"dev@dpdk.org" <dev@dpdk.org>
Cc: "ayeh@cisco.com" <ayeh@cisco.com>,
"Stokes, Ian" <ian.stokes@intel.com>,
"yega@cisco.com" <yega@cisco.com>,
"Bendror, Eran (Nokia - US)" <eran.bendror@nokia.com>
Subject: Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service
Date: Mon, 21 Feb 2022 15:19:50 +0100 [thread overview]
Message-ID: <85935ffc-aa30-6f08-df02-94dccf394872@suse.com> (raw)
In-Reply-To: <SA1PR08MB7133FD67159515D95D1A11DA83639@SA1PR08MB7133.namprd08.prod.outlook.com>
Hello,
I have been seeing the same issue with several different DPDK-OVS
versions as well as QEMU versions.
It looks like an issue with handling the VHOST_USER_GET_VRING_BASE once
the application in the guest is restarted. It might probably have to do
with QEMU asynchronous message passing...
I am not an expert on the vhost/virtio so trying to have your help with
this. Has anybody had the chance to look into this issue and found a
solution or workaround?
Cheers,
Marco
On 11/26/21 15:09, Bendror, Eran (Nokia - US) wrote:
> Hi,
>
> Internally the VM is using DPDK 17.05, on Centos7.9 – but this seems to
> be reproducing with guest level 18.11 as well.
>
> The issue is when the DPDK PMDs get started at guest, so the assumption
> is that that presents bad / inaccessible memory towards the host.
>
> We did notice some mis-use at the guest of selinux permissions, and
> removing that helped reducing the frequency significantly.
>
> Is there a way to map the shared memory between VM and host to see where
> is the segmentation fault coming from?
>
> I will see if I can upload the VM xml, but it is a multi-queue 4 port VM.
>
> Thanks for the assistance,
>
> Eran
>
> *From:* Xia, Chenbo <chenbo.xia@intel.com>
> *Sent:* Friday, November 26, 2021 4:25 AM
> *To:* Bendror, Eran (Nokia - US) <eran.bendror@nokia.com>;
> ktraynor@redhat.com
> *Cc:* ayeh@cisco.com; dev@dpdk.org; Stokes, Ian <ian.stokes@intel.com>;
> maxime.coquelin@redhat.com; yega@cisco.com; Marco Varlese
> <marco.varlese@suse.com>
> *Subject:* RE: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when
> guest VM restarts network service
>
> Hi,
>
> Is it possible that you can provide more info about this isuee. I mean:
> qemu cmdline/libvirt xml, ovs cmdline, guest driver version and etc… Or
> it’s hard to reproduce the issue.
>
> Thanks,
>
> Chenbo
>
> *From:* Bendror, Eran (Nokia - US) <eran.bendror@nokia.com
> <mailto:eran.bendror@nokia.com>>
> *Sent:* Wednesday, November 17, 2021 10:42 PM
> *To:* ktraynor@redhat.com <mailto:ktraynor@redhat.com>
> *Cc:* ayeh@cisco.com <mailto:ayeh@cisco.com>; Xia, Chenbo
> <chenbo.xia@intel.com <mailto:chenbo.xia@intel.com>>; dev@dpdk.org
> <mailto:dev@dpdk.org>; Stokes, Ian <ian.stokes@intel.com
> <mailto:ian.stokes@intel.com>>; maxime.coquelin@redhat.com
> <mailto:maxime.coquelin@redhat.com>; yega@cisco.com <mailto:yega@cisco.com>
> *Subject:* Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when
> guest VM restarts network service
>
> Hello,
>
> I am wondering if there was any progress in this topic, we are seeing a
> very similar issue, where a VM level application restart triggers
> segmentation fault and failed to allocate MBuf on the host level
>
> CentOS Linux release 7.8.2003 (Core)
>
> dpdk-18.11.5-1.el7_8.x86_64
>
> openvswitch-2.11.0-4.el7.x86_64
>
> libvirt 4.5.0
>
> QEMU 4.5.0 (API)
>
> QEMU 2.12.0
>
> 3.10.0-1127.13.1.el7.x86_64
>
> And we get the same crash
>
> #0 0x00007f96cb72e7ee in rte_memcpy_generic () from
> /lib64/librte_vhost.so.4
>
> #1 0x00007f96cb7350f2 in rte_vhost_dequeue_burst () from
> /lib64/librte_vhost.so.4
>
> #2 0x00007f96caf97f03 in netdev_dpdk_vhost_rxq_recv () from
> /lib64/libopenvswitch-2.11.so.0
>
> #3 0x00007f96caed21e6 in netdev_rxq_recv () from
> /lib64/libopenvswitch-2.11.so.0
>
> #4 0x00007f96caea07ca in dp_netdev_process_rxq_port () from
> /lib64/libopenvswitch-2.11.so.0
>
> #5 0x00007f96caea0ca5 in pmd_thread_main () from
> /lib64/libopenvswitch-2.11.so.0
>
> #6 0x00007f96caf2da3f in ovsthread_wrapper () from
> /lib64/libopenvswitch-2.11.so.0
>
> #7 0x00007f96c9ef3ea5 in start_thread () from /lib64/libpthread.so.0
>
> #8 0x00007f96c94118dd in clone () from /lib64/libc.so.6
>
> We have tried upgrading host level artifacts:
>
> dpdk-20.11.3-1.el7.x86_64
>
> openvswitch-2.16.1-1.el7.x86_64
>
> With backtrace:
>
> #0 0x00007f6b8b49748c in virtio_dev_tx_split_legacy () from
> /lib64/librte_vhost.so.21
>
> #1 0x00007f6b8b4c0fdb in rte_vhost_dequeue_burst () from
> /lib64/librte_vhost.so.21
>
> #2 0x000055bd714c2802 in netdev_dpdk_vhost_rxq_recv ()
>
> #3 0x000055bd713f8e51 in netdev_rxq_recv ()
>
> #4 0x000055bd713c9d2a in dp_netdev_process_rxq_port ()
>
> #5 0x000055bd713ca1f9 in pmd_thread_main ()
>
> #6 0x000055bd71455cdf in ovsthread_wrapper ()
>
> #7 0x00007f6b8a6a9ea5 in start_thread () from /lib64/libpthread.so.0
>
> #8 0x00007f6b89bc78dd in clone () from /lib64/libc.so.6
>
> Regards,
>
> Eran
>
next prev parent reply other threads:[~2022-02-21 14:19 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-17 14:42 Bendror, Eran (Nokia - US)
2021-11-26 9:24 ` Xia, Chenbo
2021-11-26 14:09 ` Bendror, Eran (Nokia - US)
2022-02-21 14:19 ` Marco Varlese [this message]
-- strict thread matches above, loose matches on Subject: below --
2020-11-19 1:44 [dpdk-dev] " Alex Yeh (ayeh)
2020-11-19 11:21 ` [dpdk-dev] [ovs-dev] " Stokes, Ian
2020-11-19 12:08 ` Kevin Traynor
2021-01-08 19:35 ` Alex Yeh (ayeh)
2021-01-12 18:20 ` Alex Yeh (ayeh)
2021-01-13 14:14 ` Kevin Traynor
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=85935ffc-aa30-6f08-df02-94dccf394872@suse.com \
--to=marco.varlese@suse.com \
--cc=ayeh@cisco.com \
--cc=chenbo.xia@intel.com \
--cc=dev@dpdk.org \
--cc=eran.bendror@nokia.com \
--cc=ian.stokes@intel.com \
--cc=ktraynor@redhat.com \
--cc=maxime.coquelin@redhat.com \
--cc=yega@cisco.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).