DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Bendror, Eran (Nokia - US)" <eran.bendror@nokia.com>
To: "Xia, Chenbo" <chenbo.xia@intel.com>,
	"ktraynor@redhat.com" <ktraynor@redhat.com>
Cc: "ayeh@cisco.com" <ayeh@cisco.com>, "dev@dpdk.org" <dev@dpdk.org>,
	"Stokes,  Ian" <ian.stokes@intel.com>,
	"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
	"yega@cisco.com" <yega@cisco.com>,
	Marco Varlese <marco.varlese@suse.com>
Subject: RE: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service
Date: Fri, 26 Nov 2021 14:09:48 +0000
Message-ID: <SA1PR08MB7133FD67159515D95D1A11DA83639@SA1PR08MB7133.namprd08.prod.outlook.com> (raw)
In-Reply-To: <SN6PR11MB3504571DB2BFB31425D9CDB49C639@SN6PR11MB3504.namprd11.prod.outlook.com>

[-- Attachment #1: Type: text/plain, Size: 3729 bytes --]

Hi,

Internally the VM is using DPDK 17.05, on Centos7.9 – but this seems to be reproducing with guest level 18.11 as well.

The issue is when the DPDK PMDs get started at guest, so the assumption is that that presents bad / inaccessible memory towards the host.

We did notice some mis-use at the guest of selinux permissions, and removing that helped reducing the frequency significantly.

Is there a way to map the shared memory between VM and host to see where is the segmentation fault coming from?

I will see if I can upload the VM xml, but it is a multi-queue 4 port VM.

Thanks for the assistance,
Eran

From: Xia, Chenbo <chenbo.xia@intel.com>
Sent: Friday, November 26, 2021 4:25 AM
To: Bendror, Eran (Nokia - US) <eran.bendror@nokia.com>; ktraynor@redhat.com
Cc: ayeh@cisco.com; dev@dpdk.org; Stokes, Ian <ian.stokes@intel.com>; maxime.coquelin@redhat.com; yega@cisco.com; Marco Varlese <marco.varlese@suse.com>
Subject: RE: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service

Hi,

Is it possible that you can provide more info about this isuee. I mean: qemu cmdline/libvirt xml, ovs cmdline, guest driver version and etc… Or it’s hard to reproduce the issue.

Thanks,
Chenbo

From: Bendror, Eran (Nokia - US) <eran.bendror@nokia.com<mailto:eran.bendror@nokia.com>>
Sent: Wednesday, November 17, 2021 10:42 PM
To: ktraynor@redhat.com<mailto:ktraynor@redhat.com>
Cc: ayeh@cisco.com<mailto:ayeh@cisco.com>; Xia, Chenbo <chenbo.xia@intel.com<mailto:chenbo.xia@intel.com>>; dev@dpdk.org<mailto:dev@dpdk.org>; Stokes, Ian <ian.stokes@intel.com<mailto:ian.stokes@intel.com>>; maxime.coquelin@redhat.com<mailto:maxime.coquelin@redhat.com>; yega@cisco.com<mailto:yega@cisco.com>
Subject: Re: [dpdk-dev] [ovs-dev] ovs-vswitchd with DPDK crashed when guest VM restarts network service

Hello,

I am wondering if there was any progress in this topic, we are seeing a very similar issue, where a VM level application restart triggers segmentation fault and failed to allocate MBuf on the host level

CentOS Linux release 7.8.2003 (Core)
dpdk-18.11.5-1.el7_8.x86_64
openvswitch-2.11.0-4.el7.x86_64
libvirt 4.5.0
QEMU 4.5.0 (API)
QEMU 2.12.0
3.10.0-1127.13.1.el7.x86_64

And we get the same crash

#0  0x00007f96cb72e7ee in rte_memcpy_generic () from /lib64/librte_vhost.so.4
#1  0x00007f96cb7350f2 in rte_vhost_dequeue_burst () from /lib64/librte_vhost.so.4
#2  0x00007f96caf97f03 in netdev_dpdk_vhost_rxq_recv () from /lib64/libopenvswitch-2.11.so.0
#3  0x00007f96caed21e6 in netdev_rxq_recv () from /lib64/libopenvswitch-2.11.so.0
#4  0x00007f96caea07ca in dp_netdev_process_rxq_port () from /lib64/libopenvswitch-2.11.so.0
#5  0x00007f96caea0ca5 in pmd_thread_main () from /lib64/libopenvswitch-2.11.so.0
#6  0x00007f96caf2da3f in ovsthread_wrapper () from /lib64/libopenvswitch-2.11.so.0
#7  0x00007f96c9ef3ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f96c94118dd in clone () from /lib64/libc.so.6

We have tried upgrading host level artifacts:

dpdk-20.11.3-1.el7.x86_64
openvswitch-2.16.1-1.el7.x86_64

With backtrace:

#0  0x00007f6b8b49748c in virtio_dev_tx_split_legacy () from /lib64/librte_vhost.so.21
#1  0x00007f6b8b4c0fdb in rte_vhost_dequeue_burst () from /lib64/librte_vhost.so.21
#2  0x000055bd714c2802 in netdev_dpdk_vhost_rxq_recv ()
#3  0x000055bd713f8e51 in netdev_rxq_recv ()
#4  0x000055bd713c9d2a in dp_netdev_process_rxq_port ()
#5  0x000055bd713ca1f9 in pmd_thread_main ()
#6  0x000055bd71455cdf in ovsthread_wrapper ()
#7  0x00007f6b8a6a9ea5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f6b89bc78dd in clone () from /lib64/libc.so.6

Regards,
Eran


[-- Attachment #2: Type: text/html, Size: 8656 bytes --]

  reply	other threads:[~2021-11-28 12:56 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-17 14:42 Bendror, Eran (Nokia - US)
2021-11-26  9:24 ` Xia, Chenbo
2021-11-26 14:09   ` Bendror, Eran (Nokia - US) [this message]
2022-02-21 14:19     ` Marco Varlese
  -- strict thread matches above, loose matches on Subject: below --
2020-11-19  1:44 [dpdk-dev] " Alex Yeh (ayeh)
2020-11-19 11:21 ` [dpdk-dev] [ovs-dev] " Stokes, Ian
2020-11-19 12:08   ` Kevin Traynor
2021-01-08 19:35     ` Alex Yeh (ayeh)
2021-01-12 18:20       ` Alex Yeh (ayeh)
2021-01-13 14:14         ` Kevin Traynor

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=SA1PR08MB7133FD67159515D95D1A11DA83639@SA1PR08MB7133.namprd08.prod.outlook.com \
    --to=eran.bendror@nokia.com \
    --cc=ayeh@cisco.com \
    --cc=chenbo.xia@intel.com \
    --cc=dev@dpdk.org \
    --cc=ian.stokes@intel.com \
    --cc=ktraynor@redhat.com \
    --cc=marco.varlese@suse.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=yega@cisco.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git