From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f54.google.com (mail-wm0-f54.google.com [74.125.82.54]) by dpdk.org (Postfix) with ESMTP id 3929EFFA for ; Mon, 6 Feb 2017 20:42:21 +0100 (CET) Received: by mail-wm0-f54.google.com with SMTP id r141so127842249wmg.1 for ; Mon, 06 Feb 2017 11:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=u+S74sEGmYPF5c+LeTj+YdaNt8zPbS6mPwjJ9iw/a5Q=; b=uZRXHC6ThA0Ax0eACqGHAxlgu+PKX+ltrlagievgOMPpRyfnxCQz3zlaIDnIDpuLKn X0DzKWjVa0xLfaIhBbw+sdme/RiRHmEhEyOXfwdN8cgOrWDWqLmY2KzRSrEeakSSgh0h u+qBwJ8Ub63FZlcX5e0aEDiVWSeg0iVAjdvZQCI2F67ZXRK4+cNjosYgTYVQxh+Jyv52 nipS/ziWaTcAMknRHH3ygfC0U0ZG0YjEKpRS4jmQgiJqAdwH+erQkpjMMUz0eWZcw50m qQi5XvH3FMh16ZnsNxRkJn7DtocuzzPJwMeCEd6P6RlfMwvU2YJjTHxKdJFRiDU8GuT8 x60g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=u+S74sEGmYPF5c+LeTj+YdaNt8zPbS6mPwjJ9iw/a5Q=; b=ZA5Q6mWKTO3PJDjjnjILEHXNWJjP/M5bPRUoUdxNYbrBMMoTeKzq9DqKsewKnuCWvD 1xUXlO4uU2TXGnk+icGYw1ltGij0GWSmWcs0pKFJFcKfnzwubZK7eIAlYE0Zq4rZ7RtW abzhmy0YDY293uaJu/1sNtGOR/1GXjs9MfFuUbZ4hg+W8k1WWN8HJzcWEkpO+Sxbz0gh +CjZqejpDR5bWj97x9DKuQfCFimGfZjIzdnlDMYOD9kR9BNysIIwQJVC0wDUUgOAUy4v MqZTNezCi7AKV/7QcLcR4cVHjHv7lAEaovDQ1AxmGU7jj9wYFdlaAZqNQXy0Blg08C/e Lwrw== X-Gm-Message-State: AMke39nxKVfcQgjLgVFhsA4/C9VC2eX6iZv/m2WqEfVLBa6tBDXppDf/v21py5KFpjxKgP7FXYHkCbsbhTf05w== X-Received: by 10.28.206.199 with SMTP id e190mr10325063wmg.98.1486410140356; Mon, 06 Feb 2017 11:42:20 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.17.66 with HTTP; Mon, 6 Feb 2017 11:42:19 -0800 (PST) From: Burak Demirel Date: Mon, 6 Feb 2017 22:42:19 +0300 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] OVS-DPDK Segmentation Fault X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2017 19:42:21 -0000 Hi All, We use RH 7.3 and OVS 2.5.0. From time to time we face the problem below where the OVS has a segmentation fault and the guest loses connectivity. It seems that the monitoring thread can not restart the OVS. When we manually restart the OVS, we no longer see the issue but we still need a restart of the guest to get the connectivity back. We can easily replicate this by sending a member join (mcast) msg from one of the guests to join a cluster and everytime we do that, we have this problem. We have assigned socket-mem 4096,1024 and -n 4. Any ideas on resolving this or is this something known and got fixed in the upstream? ovs-vsctl show cfd11977-ab4a-4a73-a952-4e8ed2941ded Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "ovsdpdkbr0" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "ovsdpdkbr0" Interface "ovsdpdkbr0" type: internal Port "phy-ovsdpdkbr0" Interface "phy-ovsdpdkbr0" type: patch options: {peer="int-ovsdpdkbr0"} Port "dpdkbond0" *Interface "dpdk0" type: dpdk error: "could not open network device dpdk0 (Cannot allocate memory)" Interface "dpdk3" type: dpdk error: "could not open network device dpdk3 (Cannot allocate memory)" Interface "dpdk1" type: dpdk error: "could not open network device dpdk1 (Cannot allocate memory)" * ovs-vswitchd.log: 2017-02-06T08:11:45.466Z|00287|coverage(vhost_thread1)|INFO|48 events never hit 2017-02-06T08:11:45.466Z|00288|dpdk(vhost_thread1)|INFO|vHost Device '/var/run/openvswitch/vhu67e39e1b-4f' 2 has been removed 2017-02-06T08:11:45.538Z|00289|dpdk(vhost_thread1)|INFO|vHost Device '/var/run/openvswitch/vhue6e88b22-59' 1 has been removed 2017-02-06T08:11:56.719Z|00290|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/var/run/openvswitch/vhu67e39e1b-4f' 2 changed to 'enabled' 2017-02-06T08:11:56.719Z|00291|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/var/run/openvswitch/vhue6e88b22-59' 1 changed to 'enabled' 2017-02-06T08:11:59.373Z|00292|dpdk(vhost_thread1)|INFO|vHost Device '/var/run/openvswitch/vhu67e39e1b-4f' 2 has been added 2017-02-06T08:11:59.373Z|00293|dpdk(vhost_thread1)|INFO|vHost Device '/var/run/openvswitch/vhue6e88b22-59' 1 has been added 2017-02-06T08:11:59.373Z|00294|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/var/run/openvswitch/vhu67e39e1b-4f' 2 changed to 'enabled' 2017-02-06T08:11:59.373Z|00295|dpdk(vhost_thread1)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/var/run/openvswitch/vhue6e88b22-59' 1 changed to 'enabled' *2017-02-06T08:49:08.548Z|00003|daemon_unix(monitor)|ERR|1 crashes: pid 32916 died, killed (Segmentation fault), core dumped, restarting* 2017-02-06T08:49:08.589Z|00004|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2017-02-06T08:49:08.589Z|00005|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2017-02-06T08:49:08.589Z|00006|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2017-02-06T08:49:08.589Z|00007|memory|INFO|87988 kB peak resident set size after 3079.5 seconds 2017-02-06T08:49:08.589Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2017-02-06T08:49:08.589Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2017-02-06T08:49:08.598Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation 2017-02-06T08:49:08.598Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 2017-02-06T08:49:08.598Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids 2017-02-06T08:49:08.598Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state 2017-02-06T08:49:08.598Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_zone 2017-02-06T08:49:08.598Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_mark 2017-02-06T08:49:08.598Z|00016|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_label 2017-02-06T08:49:08.639Z|00017|bridge|INFO|bridge ovsdpdkbr0: added interface ovsdpdkbr0 on port 65534 2017-02-06T08:49:08.639Z|00018|bridge|WARN|could not open network device dpdk3 (Cannot allocate memory) 2017-02-06T08:49:08.639Z|00019|bridge|WARN|could not open network device dpdk0 (Cannot allocate memory) 2017-02-06T08:49:08.639Z|00020|bridge|WARN|could not open network device dpdk1 (Cannot allocate memory) 2017-02-06T08:49:08.639Z|00021|bridge|INFO|bridge ovsdpdkbr0: added interface phy-ovsdpdkbr0 on port 4 2017-02-06T08:49:08.639Z|00022|bridge|INFO|bridge br-int: added interface int-ovsdpdkbr0 on port 7 2017-02-06T08:49:08.639Z|00023|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue6e88b22-59 2017-02-06T08:49:08.639Z|00024|bridge|WARN|could not open network device vhue6e88b22-59 (Unknown error -1) 2017-02-06T08:49:08.639Z|00025|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue1fe2f9a-29 2017-02-06T08:49:08.639Z|00026|bridge|WARN|could not open network device vhue1fe2f9a-29 (Unknown error -1) 2017-02-06T08:49:08.639Z|00027|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhubd9d8402-d6 2017-02-06T08:49:08.639Z|00028|bridge|WARN|could not open network device vhubd9d8402-d6 (Unknown error -1) 2017-02-06T08:49:08.639Z|00029|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue6013e82-f7 2017-02-06T08:49:08.639Z|00030|bridge|WARN|could not open network device vhue6013e82-f7 (Unknown error -1) 2017-02-06T08:49:08.639Z|00031|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu8d589307-cc 2017-02-06T08:49:08.639Z|00032|bridge|WARN|could not open network device vhu8d589307-cc (Unknown error -1) 2017-02-06T08:49:08.639Z|00033|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu949db271-aa 2017-02-06T08:49:08.639Z|00034|bridge|WARN|could not open network device vhu949db271-aa (Unknown error -1) 2017-02-06T08:49:08.639Z|00035|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu0f0f88cd-a8 2017-02-06T08:49:08.639Z|00036|bridge|WARN|could not open network device vhu0f0f88cd-a8 (Unknown error -1) 2017-02-06T08:49:08.640Z|00037|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu600b1c3c-47 2017-02-06T08:49:08.640Z|00038|bridge|WARN|could not open network device vhu600b1c3c-47 (Unknown error -1) 2017-02-06T08:49:08.640Z|00039|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhuf72a5c18-95 2017-02-06T08:49:08.640Z|00040|bridge|WARN|could not open network device vhuf72a5c18-95 (Unknown error -1) 2017-02-06T08:49:08.640Z|00041|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhua39fa843-c3 2017-02-06T08:49:08.640Z|00042|bridge|WARN|could not open network device vhua39fa843-c3 (Unknown error -1) 2017-02-06T08:49:08.640Z|00043|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhude918d08-ec 2017-02-06T08:49:08.640Z|00044|bridge|WARN|could not open network device vhude918d08-ec (Unknown error -1) 2017-02-06T08:49:08.640Z|00045|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu13d398ba-fc 2017-02-06T08:49:08.640Z|00046|bridge|WARN|could not open network device vhu13d398ba-fc (Unknown error -1) 2017-02-06T08:49:08.640Z|00047|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhub51d2503-42 2017-02-06T08:49:08.640Z|00048|bridge|WARN|could not open network device vhub51d2503-42 (Unknown error -1) 2017-02-06T08:49:08.640Z|00049|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu6dbc3341-c9 2017-02-06T08:49:08.640Z|00050|bridge|WARN|could not open network device vhu6dbc3341-c9 (Unknown error -1) 2017-02-06T08:49:08.640Z|00051|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhua4ab858d-e5 2017-02-06T08:49:08.640Z|00052|bridge|WARN|could not open network device vhua4ab858d-e5 (Unknown error -1) 2017-02-06T08:49:08.640Z|00053|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu554d961a-6b 2017-02-06T08:49:08.640Z|00054|bridge|WARN|could not open network device vhu554d961a-6b (Unknown error -1) 2017-02-06T08:49:08.640Z|00055|bridge|INFO|bridge br-int: added interface br-int on port 65534 2017-02-06T08:49:08.641Z|00056|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu67e39e1b-4f 2017-02-06T08:49:08.641Z|00057|bridge|WARN|could not open network device vhu67e39e1b-4f (Unknown error -1) 2017-02-06T08:49:08.641Z|00058|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhufb7d1a45-2f 2017-02-06T08:49:08.641Z|00059|bridge|WARN|could not open network device vhufb7d1a45-2f (Unknown error -1) 2017-02-06T08:49:08.641Z|00060|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhu00e22fb6-ed 2017-02-06T08:49:08.641Z|00061|bridge|WARN|could not open network device vhu00e22fb6-ed (Unknown error -1) 2017-02-06T08:49:08.641Z|00062|bridge|INFO|bridge ovsdpdkbr0: using datapath ID 000002540980bf41 2017-02-06T08:49:08.642Z|00063|connmgr|INFO|ovsdpdkbr0: added service controller "punix:/var/run/openvswitch/ovsdpdkbr0.mgmt" 2017-02-06T08:49:08.642Z|00064|connmgr|INFO|ovsdpdkbr0: added primary controller "tcp:127.0.0.1:6633" 2017-02-06T08:49:08.642Z|00065|rconn|INFO|ovsdpdkbr0<->tcp:127.0.0.1:6633: connecting... 2017-02-06T08:49:08.690Z|00066|bridge|INFO|bridge br-int: using datapath ID 0000065189af7548 2017-02-06T08:49:08.690Z|00067|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" 2017-02-06T08:49:08.690Z|00068|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" 2017-02-06T08:49:08.690Z|00069|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting... 2017-02-06T08:49:08.740Z|00070|rconn|INFO|ovsdpdkbr0<->tcp:127.0.0.1:6633: connected 2017-02-06T08:49:08.740Z|00071|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connected 2017-02-06T08:49:08.741Z|00072|bridge|WARN|could not open network device dpdk3 (Cannot allocate memory) 2017-02-06T08:49:08.741Z|00073|bridge|WARN|could not open network device dpdk0 (Cannot allocate memory) 2017-02-06T08:49:08.741Z|00074|bridge|WARN|could not open network device dpdk1 (Cannot allocate memory) 2017-02-06T08:49:08.741Z|00075|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue6e88b22-59 2017-02-06T08:49:08.741Z|00076|bridge|WARN|could not open network device vhue6e88b22-59 (Unknown error -1) 2017-02-06T08:49:08.742Z|00077|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue1fe2f9a-29 2017-02-06T08:49:08.742Z|00078|bridge|WARN|could not open network device vhue1fe2f9a-29 (Unknown error -1) 2017-02-06T08:49:08.742Z|00079|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhubd9d8402-d6 2017-02-06T08:49:08.742Z|00080|bridge|WARN|could not open network device vhubd9d8402-d6 (Unknown error -1) 2017-02-06T08:49:08.742Z|00081|dpdk|ERR|vhost-user socket device setup failure for socket /var/run/openvswitch/vhue6013e82-f7 some other logs from another crash time Feb 6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to allocate memory for mbuf. Feb 6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to allocate memory for mbuf. Feb 6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to allocate memory for mbuf. Feb 6 10:36:32 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to allocate memory for mbuf. Feb 6 10:36:33 compute-06 ovs-vswitchd[3122]: VHOST_DATA: Failed to allocate memory for mbuf. Feb 6 10:36:33 compute-06 kernel: vhost_thread1[3124]: segfault at 20 ip 00007f6fc6e753b7 sp 00007f6fc21185d8 error 4 in libc-2.17.so [7f6fc6d41000+1b6000] Feb 6 10:36:34 compute-06 ovs-vswitchd[3121]: ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 3122 died, killed (Segmentation fault), core dumped, restarting Mon 2017-02-06 10:36:34.134521 +03 [s=339299f7b2f14e9490e6f7d0c4652709;i=1415;b=281e7a2d64be4faa8e1823110f040ed3;m=f1c74de56;t=547d7b0d97330;x=d9afa675780dc6bb] _UID=0 _SYSTEMD_SLICE=system.slice _BOOT_ID=281e7a2d64be4faa8e1823110f040ed3 _MACHINE_ID=29b9358ef5a34f78b9c6a781c33ee5b4 PRIORITY=3 SYSLOG_FACILITY=3 _CAP_EFFECTIVE=1fffffffff _HOSTNAME=compute-06.turkcell.com.tr _TRANSPORT=syslog _GID=107 _SYSTEMD_CGROUP=/system.slice/openvswitch-nonetwork.service _SYSTEMD_UNIT=openvswitch-nonetwork.service _SELINUX_CONTEXT=system_u:system_r:openvswitch_t:s0 SYSLOG_IDENTIFIER=ovs-vswitchd _EXE=/usr/sbin/ovs-vswitchd SYSLOG_PID=3121 MESSAGE=ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid 3122 died, killed (Segmentation fault), core dumped, restarting _PID=3121 _COMM=monitor _CMDLINE=ovs-vswit... --dpdk -l 1,2,3,25,26,27,13,37 -n 4 --socket-mem 4096 1024 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor _SOURCE_REALTIME_TIMESTAMP=1486366594134521 thanks, regards -- BBD