DPDK usage discussions
 help / color / mirror / Atom feed
* [dpdk-users] OVS-DPDK Segmentation Fault
@ 2017-02-06 19:42 Burak Demirel
  0 siblings, 0 replies; only message in thread
From: Burak Demirel @ 2017-02-06 19:42 UTC (permalink / raw)
  To: users

Hi All,

We use RH 7.3 and OVS 2.5.0. From time to time we face the problem below
where the OVS has a segmentation fault and the guest loses connectivity. It
seems that the monitoring thread can not restart the OVS. When we manually
restart the OVS, we no longer see the issue but we still need a restart of
the guest to get the connectivity back. We can easily replicate this by
sending a member join (mcast) msg from one of the guests to join a cluster
and everytime we do that, we have this problem.

We have assigned socket-mem 4096,1024 and -n 4.  Any ideas on resolving
this or is this something known and got fixed in the upstream?

ovs-vsctl show
cfd11977-ab4a-4a73-a952-4e8ed2941ded
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge "ovsdpdkbr0"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "ovsdpdkbr0"
            Interface "ovsdpdkbr0"
                type: internal
        Port "phy-ovsdpdkbr0"
            Interface "phy-ovsdpdkbr0"
                type: patch
                options: {peer="int-ovsdpdkbr0"}
        Port "dpdkbond0"










*Interface "dpdk0"                type: dpdk                error: "could
not open network device dpdk0 (Cannot allocate memory)"
Interface "dpdk3"                type: dpdk                error: "could
not open network device dpdk3 (Cannot allocate memory)"
Interface "dpdk1"                type: dpdk                error: "could
not open network device dpdk1 (Cannot allocate memory)" *
ovs-vswitchd.log:

2017-02-06T08:11:45.466Z|00287|coverage(vhost_thread1)|INFO|48 events never
hit
2017-02-06T08:11:45.466Z|00288|dpdk(vhost_thread1)|INFO|vHost Device
'/var/run/openvswitch/vhu67e39e1b-4f' 2 has been removed
2017-02-06T08:11:45.538Z|00289|dpdk(vhost_thread1)|INFO|vHost Device
'/var/run/openvswitch/vhue6e88b22-59' 1 has been removed
2017-02-06T08:11:56.719Z|00290|dpdk(vhost_thread1)|INFO|State of queue 0 (
tx_qid 0 ) of vhost device '/var/run/openvswitch/vhu67e39e1b-4f' 2 changed
to 'enabled'
2017-02-06T08:11:56.719Z|00291|dpdk(vhost_thread1)|INFO|State of queue 0 (
tx_qid 0 ) of vhost device '/var/run/openvswitch/vhue6e88b22-59' 1 changed
to 'enabled'
2017-02-06T08:11:59.373Z|00292|dpdk(vhost_thread1)|INFO|vHost Device
'/var/run/openvswitch/vhu67e39e1b-4f' 2 has been added
2017-02-06T08:11:59.373Z|00293|dpdk(vhost_thread1)|INFO|vHost Device
'/var/run/openvswitch/vhue6e88b22-59' 1 has been added
2017-02-06T08:11:59.373Z|00294|dpdk(vhost_thread1)|INFO|State of queue 0 (
tx_qid 0 ) of vhost device '/var/run/openvswitch/vhu67e39e1b-4f' 2 changed
to 'enabled'
2017-02-06T08:11:59.373Z|00295|dpdk(vhost_thread1)|INFO|State of queue 0 (
tx_qid 0 ) of vhost device '/var/run/openvswitch/vhue6e88b22-59' 1 changed
to 'enabled'
*2017-02-06T08:49:08.548Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid
32916 died, killed (Segmentation fault), core dumped, restarting*
2017-02-06T08:49:08.589Z|00004|ovs_numa|INFO|Discovered 24 CPU cores on
NUMA node 0
2017-02-06T08:49:08.589Z|00005|ovs_numa|INFO|Discovered 24 CPU cores on
NUMA node 1
2017-02-06T08:49:08.589Z|00006|ovs_numa|INFO|Discovered 2 NUMA nodes and 48
CPU cores
2017-02-06T08:49:08.589Z|00007|memory|INFO|87988 kB peak resident set size
after 3079.5 seconds
2017-02-06T08:49:08.589Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connecting...
2017-02-06T08:49:08.589Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
connected
2017-02-06T08:49:08.598Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports recirculation
2017-02-06T08:49:08.598Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS
label stack length probed as 3
2017-02-06T08:49:08.598Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath supports unique flow ids
2017-02-06T08:49:08.598Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_state
2017-02-06T08:49:08.598Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_zone
2017-02-06T08:49:08.598Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_mark
2017-02-06T08:49:08.598Z|00016|ofproto_dpif|INFO|netdev@ovs-netdev:
Datapath does not support ct_label
2017-02-06T08:49:08.639Z|00017|bridge|INFO|bridge ovsdpdkbr0: added
interface ovsdpdkbr0 on port 65534
2017-02-06T08:49:08.639Z|00018|bridge|WARN|could not open network device
dpdk3 (Cannot allocate memory)
2017-02-06T08:49:08.639Z|00019|bridge|WARN|could not open network device
dpdk0 (Cannot allocate memory)
2017-02-06T08:49:08.639Z|00020|bridge|WARN|could not open network device
dpdk1 (Cannot allocate memory)
2017-02-06T08:49:08.639Z|00021|bridge|INFO|bridge ovsdpdkbr0: added
interface phy-ovsdpdkbr0 on port 4
2017-02-06T08:49:08.639Z|00022|bridge|INFO|bridge br-int: added interface
int-ovsdpdkbr0 on port 7
2017-02-06T08:49:08.639Z|00023|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhue6e88b22-59
2017-02-06T08:49:08.639Z|00024|bridge|WARN|could not open network device
vhue6e88b22-59 (Unknown error -1)
2017-02-06T08:49:08.639Z|00025|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhue1fe2f9a-29
2017-02-06T08:49:08.639Z|00026|bridge|WARN|could not open network device
vhue1fe2f9a-29 (Unknown error -1)
2017-02-06T08:49:08.639Z|00027|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhubd9d8402-d6
2017-02-06T08:49:08.639Z|00028|bridge|WARN|could not open network device
vhubd9d8402-d6 (Unknown error -1)
2017-02-06T08:49:08.639Z|00029|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhue6013e82-f7
2017-02-06T08:49:08.639Z|00030|bridge|WARN|could not open network device
vhue6013e82-f7 (Unknown error -1)
2017-02-06T08:49:08.639Z|00031|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu8d589307-cc
2017-02-06T08:49:08.639Z|00032|bridge|WARN|could not open network device
vhu8d589307-cc (Unknown error -1)
2017-02-06T08:49:08.639Z|00033|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu949db271-aa
2017-02-06T08:49:08.639Z|00034|bridge|WARN|could not open network device
vhu949db271-aa (Unknown error -1)
2017-02-06T08:49:08.639Z|00035|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu0f0f88cd-a8
2017-02-06T08:49:08.639Z|00036|bridge|WARN|could not open network device
vhu0f0f88cd-a8 (Unknown error -1)
2017-02-06T08:49:08.640Z|00037|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu600b1c3c-47
2017-02-06T08:49:08.640Z|00038|bridge|WARN|could not open network device
vhu600b1c3c-47 (Unknown error -1)
2017-02-06T08:49:08.640Z|00039|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhuf72a5c18-95
2017-02-06T08:49:08.640Z|00040|bridge|WARN|could not open network device
vhuf72a5c18-95 (Unknown error -1)
2017-02-06T08:49:08.640Z|00041|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhua39fa843-c3
2017-02-06T08:49:08.640Z|00042|bridge|WARN|could not open network device
vhua39fa843-c3 (Unknown error -1)
2017-02-06T08:49:08.640Z|00043|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhude918d08-ec
2017-02-06T08:49:08.640Z|00044|bridge|WARN|could not open network device
vhude918d08-ec (Unknown error -1)
2017-02-06T08:49:08.640Z|00045|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu13d398ba-fc
2017-02-06T08:49:08.640Z|00046|bridge|WARN|could not open network device
vhu13d398ba-fc (Unknown error -1)
2017-02-06T08:49:08.640Z|00047|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhub51d2503-42
2017-02-06T08:49:08.640Z|00048|bridge|WARN|could not open network device
vhub51d2503-42 (Unknown error -1)
2017-02-06T08:49:08.640Z|00049|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu6dbc3341-c9
2017-02-06T08:49:08.640Z|00050|bridge|WARN|could not open network device
vhu6dbc3341-c9 (Unknown error -1)
2017-02-06T08:49:08.640Z|00051|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhua4ab858d-e5
2017-02-06T08:49:08.640Z|00052|bridge|WARN|could not open network device
vhua4ab858d-e5 (Unknown error -1)
2017-02-06T08:49:08.640Z|00053|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu554d961a-6b
2017-02-06T08:49:08.640Z|00054|bridge|WARN|could not open network device
vhu554d961a-6b (Unknown error -1)
2017-02-06T08:49:08.640Z|00055|bridge|INFO|bridge br-int: added interface
br-int on port 65534
2017-02-06T08:49:08.641Z|00056|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu67e39e1b-4f
2017-02-06T08:49:08.641Z|00057|bridge|WARN|could not open network device
vhu67e39e1b-4f (Unknown error -1)
2017-02-06T08:49:08.641Z|00058|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhufb7d1a45-2f
2017-02-06T08:49:08.641Z|00059|bridge|WARN|could not open network device
vhufb7d1a45-2f (Unknown error -1)
2017-02-06T08:49:08.641Z|00060|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhu00e22fb6-ed
2017-02-06T08:49:08.641Z|00061|bridge|WARN|could not open network device
vhu00e22fb6-ed (Unknown error -1)
2017-02-06T08:49:08.641Z|00062|bridge|INFO|bridge ovsdpdkbr0: using
datapath ID 000002540980bf41
2017-02-06T08:49:08.642Z|00063|connmgr|INFO|ovsdpdkbr0: added service
controller "punix:/var/run/openvswitch/ovsdpdkbr0.mgmt"
2017-02-06T08:49:08.642Z|00064|connmgr|INFO|ovsdpdkbr0: added primary
controller "tcp:127.0.0.1:6633"
2017-02-06T08:49:08.642Z|00065|rconn|INFO|ovsdpdkbr0<->tcp:127.0.0.1:6633:
connecting...
2017-02-06T08:49:08.690Z|00066|bridge|INFO|bridge br-int: using datapath ID
0000065189af7548
2017-02-06T08:49:08.690Z|00067|connmgr|INFO|br-int: added service
controller "punix:/var/run/openvswitch/br-int.mgmt"
2017-02-06T08:49:08.690Z|00068|connmgr|INFO|br-int: added primary
controller "tcp:127.0.0.1:6633"
2017-02-06T08:49:08.690Z|00069|rconn|INFO|br-int<->tcp:127.0.0.1:6633:
connecting...
2017-02-06T08:49:08.740Z|00070|rconn|INFO|ovsdpdkbr0<->tcp:127.0.0.1:6633:
connected
2017-02-06T08:49:08.740Z|00071|rconn|INFO|br-int<->tcp:127.0.0.1:6633:
connected
2017-02-06T08:49:08.741Z|00072|bridge|WARN|could not open network device
dpdk3 (Cannot allocate memory)
2017-02-06T08:49:08.741Z|00073|bridge|WARN|could not open network device
dpdk0 (Cannot allocate memory)
2017-02-06T08:49:08.741Z|00074|bridge|WARN|could not open network device
dpdk1 (Cannot allocate memory)
2017-02-06T08:49:08.741Z|00075|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhue6e88b22-59
2017-02-06T08:49:08.741Z|00076|bridge|WARN|could not open network device
vhue6e88b22-59 (Unknown error -1)
2017-02-06T08:49:08.742Z|00077|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhue1fe2f9a-29
2017-02-06T08:49:08.742Z|00078|bridge|WARN|could not open network device
vhue1fe2f9a-29 (Unknown error -1)
2017-02-06T08:49:08.742Z|00079|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhubd9d8402-d6
2017-02-06T08:49:08.742Z|00080|bridge|WARN|could not open network device
vhubd9d8402-d6 (Unknown error -1)
2017-02-06T08:49:08.742Z|00081|dpdk|ERR|vhost-user socket device setup
failure for socket /var/run/openvswitch/vhue6013e82-f7


some other logs from another crash time

Feb  6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to
allocate memory for mbuf.
Feb  6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to
allocate memory for mbuf.
Feb  6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to
allocate memory for mbuf.
Feb  6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to
allocate memory for mbuf.
Feb  6 10:36:33 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to
allocate memory for mbuf.
Feb  6 10:36:33 compute-06 kernel: vhost_thread1[3124]: segfault at 20 ip
00007f6fc6e753b7 sp 00007f6fc21185d8 error 4 in libc-2.17.so
[7f6fc6d41000+1b6000]
Feb  6 10:36:34 compute-06 ovs-vswitchd[3121]:
ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 3122 died, killed
(Segmentation fault), core dumped, restarting


Mon 2017-02-06 10:36:34.134521 +03
[s=339299f7b2f14e9490e6f7d0c4652709;i=1415;b=281e7a2d64be4faa8e1823110f040ed3;m=f1c74de56;t=547d7b0d97330;x=d9afa675780dc6bb]
    _UID=0
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=281e7a2d64be4faa8e1823110f040ed3
    _MACHINE_ID=29b9358ef5a34f78b9c6a781c33ee5b4
    PRIORITY=3
    SYSLOG_FACILITY=3
    _CAP_EFFECTIVE=1fffffffff
    _HOSTNAME=compute-06.turkcell.com.tr
    _TRANSPORT=syslog
    _GID=107
    _SYSTEMD_CGROUP=/system.slice/openvswitch-nonetwork.service
    _SYSTEMD_UNIT=openvswitch-nonetwork.service
    _SELINUX_CONTEXT=system_u:system_r:openvswitch_t:s0
    SYSLOG_IDENTIFIER=ovs-vswitchd
    _EXE=/usr/sbin/ovs-vswitchd
    SYSLOG_PID=3121
    MESSAGE=ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 3122 died,
killed (Segmentation fault), core dumped, restarting
    _PID=3121
    _COMM=monitor
    _CMDLINE=ovs-vswit... --dpdk -l 1,2,3,25,26,27,13,37 -n 4 --socket-mem
4096 1024 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err
-vfile:info --mlockall --no-chdir
--log-file=/var/log/openvswitch/ovs-vswitchd.log
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
    _SOURCE_REALTIME_TIMESTAMP=1486366594134521

thanks, regards
-- 
BBD

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2017-02-06 19:42 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-06 19:42 [dpdk-users] OVS-DPDK Segmentation Fault Burak Demirel

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).