* [dpdk-dev] ovs crash when running traffic from VM to VM over DPDK and vhostuser
[not found] ` <638383158.2812291.1459804660016.JavaMail.yahoo@mail.yahoo.com>
@ 2016-04-05 20:36 ` Yi Ba
2016-04-06 16:58 ` Yuanhan Liu
0 siblings, 1 reply; 3+ messages in thread
From: Yi Ba @ 2016-04-05 20:36 UTC (permalink / raw)
To: dev
[-- Attachment #1: Type: text/plain, Size: 2375 bytes --]
This OVS crash was first sent to openvswitch bug report mailing list, but it was suggested it is posted to dpdk as crash is in netdev code.
What you did that make the problem appear.
- We have an openstack kilo setup. it has 3 controllers and 3 computes. 1 of the controllers runs an ODL, which manages the OVS on each compute host. The compute hosts are running an hlinux OS, which is HPE's Debian8-based OS.
each host has 2 numa nodes, each with 12 cores (24 Hyper Threaded). each numa with 64GB.
We patched neutron to create vhostuser ports (which is not available in stable kilo), in order to work with dpdk in order to achieve highest throughput possible.
OVS was running with "-c 4" and pmd-core-mask 0x38. all these cores were isolated.
nova was configured with vcpu_pin_set=6-11, and the flavor had 6 vCPUs. flavor had 16 1GB huge pages, backed up by real 1GB huge pages in host.
Then running a traffic generator inside 2 VMs, using DPDK, in order to generate traffic. sending directly to the other VMs mac and IP.
- What you expected to happen.
We expected traffic to flow.
- What actually happened.
OVS crashed (in dpdk code). Attached BT.
- The Open vSwitch version number (as output by ovs-vswitchd --version)
root@BASE-CCP-CPN-N0001-NETCLM:~# ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.5.0
Compiled Apr 4 2016 08:51:09
- Any local patches or changes you have applied (if any).
applied ce179f1163f947fe8dc5afa35a2cdd0756bb53a0
The following are also handy sometimes:
- The kernel version on which Open vSwitch is running (from /proc/version) and the distribution and version number of your OS (e.g. "Centos 5.0").
root@BASE-CCP-CPN-N0001-NETCLM:~# cat /proc/version
Linux version 3.14.48-1-amd64-hlinux (pbuilder@build) (gcc version 4.9.2 (Debian 4.9.2-10) ) #hlinux1 SMP Thu Aug 6 16:02:22 UTC 2015
- If you have Open vSwitch configured to connect to an OpenFlow controller, the output of ovs-ofctl show <bridge> for each <bridge> configured in the vswitchd configuration database.
We are using odl. attached outputs.
- A fix or workaround, if you have one
We disabled mrg_rxbuf (mrg_rxbuf=off) in qemu
We can supply more info if necessary, like our exact build process etc.
[-- Attachment #2: ovs-ofctl.txt --]
[-- Type: text/plain, Size: 3641 bytes --]
root@BASE-CCP-CPN-N0001-NETCLM:~# ovs-ofctl -OOpenFlow13 show br-int
OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:0000aa08f74e9f4a
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
1(patch-ext): addr:c2:0a:3f:2e:32:e3
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(vxlan-10.54.6.1): addr:46:94:c3:b2:41:f8
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
3(vxlan-10.54.6.1): addr:16:21:e9:f5:93:b7
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
4(vxlan-10.54.6.1): addr:ba:98:7b:cd:0c:76
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
5(vxlan-10.54.6.1): addr:ca:72:68:0f:71:18
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
6(vxlan-10.54.6.1): addr:8a:6c:a3:7d:d2:96
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
7(vhu7be3fd3d-83): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
8(vhuba765032-b1): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
9(vhu646629f2-92): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
10(vhu7a4bb00f-66): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
11(vhu74d9bac3-f3): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
12(vhu06f70166-84): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-int): addr:aa:08:f7:4e:9f:4a
config: PORT_DOWN
state: LINK_DOWN
current: 10MB-FD COPPER
speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0
root@BASE-CCP-CPN-N0001-NETCLM:~# ovs-ofctl -OOpenFlow13 show br-ex
OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:000000e0ed50db4b
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
1(dpdk1): addr:00:e0:ed:50:db:4b
config: 0
state: 0
current: 10GB-FD
supported: 1GB-HD 1GB-FD 10GB-FD 1TB-FD COPPER FIBER AUTO_PAUSE AUTO_PAUSE_ASYM
speed: 10000 Mbps now, 1000000 Mbps max
2(patch-int): addr:ce:a9:49:76:97:66
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-ex): addr:00:e0:ed:50:db:4b
config: PORT_DOWN
state: LINK_DOWN
current: 10MB-FD COPPER
speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0
root@BASE-CCP-CPN-N0001-NETCLM:~# ovs-ofctl -OOpenFlow13 show br-tun
OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:000000e0ed50db4a
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
1(dpdk0): addr:00:e0:ed:50:db:4a
config: 0
state: 0
current: 10GB-FD
supported: 10GB-FD 100GB-FD 1TB-FD OTHER COPPER FIBER AUTO_NEG AUTO_PAUSE
speed: 10000 Mbps now, 1000000 Mbps max
LOCAL(br-tun): addr:00:e0:ed:50:db:4a
config: 0
state: 0
current: 10MB-FD COPPER
speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=normal miss_send_len=0
[-- Attachment #3: ovs-vswitchd-gdb.txt --]
[-- Type: text/plain, Size: 29313 bytes --]
root@BASE-CCP-CPN-N0001-NETCLM:~# gdb -p $(pidof ovs-vswitchd)
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 21152
Reading symbols from /usr/sbin/ovs-vswitchd...Reading symbols from /usr/lib/debug//usr/sbin/ovs-vswitchd...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libatomic.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libatomic.so.1
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libpthread-2.19.so...done.
done.
[New LWP 21290]
[New LWP 21289]
[New LWP 21288]
[New LWP 21287]
[New LWP 21286]
[New LWP 21285]
[New LWP 21284]
[New LWP 21283]
[New LWP 21282]
[New LWP 21281]
[New LWP 21280]
[New LWP 21279]
[New LWP 21278]
[New LWP 21277]
[New LWP 21276]
[New LWP 21275]
[New LWP 21274]
[New LWP 21273]
[New LWP 21272]
[New LWP 21271]
[New LWP 21270]
[New LWP 21269]
[New LWP 21268]
[New LWP 21267]
[New LWP 21266]
[New LWP 21265]
[New LWP 21264]
[New LWP 21263]
[New LWP 21155]
[New LWP 21154]
[New LWP 21153]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.19.so...done.
done. [30/1947]
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007ff22d399d3d in poll () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) c
Continuing.
[New Thread 0x7ff22e9ceb00 (LWP 21367)]
[Thread 0x7ff22e9ceb00 (LWP 21367) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21457)]
[Thread 0x7ff22e9ceb00 (LWP 21457) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21473)]
[Thread 0x7ff22e9ceb00 (LWP 21473) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21554)]
[Thread 0x7ff22e9ceb00 (LWP 21554) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21555)]
[Thread 0x7ff22e9ceb00 (LWP 21555) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21565)]
[Thread 0x7ff22e9ceb00 (LWP 21565) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21574)]
[Thread 0x7ff22e9ceb00 (LWP 21574) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21577)]
[Thread 0x7ff22e9ceb00 (LWP 21577) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21648)]
[Thread 0x7ff22e9ceb00 (LWP 21648) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21868)]
[Thread 0x7ff22e9ceb00 (LWP 21868) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21869)]
[Thread 0x7ff22e9ceb00 (LWP 21869) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21884)]
[Thread 0x7ff22e9ceb00 (LWP 21884) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21910)]
[Thread 0x7ff22e9ceb00 (LWP 21910) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21911)]
[Thread 0x7ff22e9ceb00 (LWP 21911) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21912)]
[Thread 0x7ff22e9ceb00 (LWP 21912) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21926)]
[Thread 0x7ff22e9ceb00 (LWP 21926) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21933)]
[Thread 0x7ff22e9ceb00 (LWP 21933) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21948)]
[Thread 0x7ff22e9ceb00 (LWP 21948) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21961)]
[Thread 0x7ff22e9ceb00 (LWP 21961) exited]
[New Thread 0x7ff22e9ceb00 (LWP 21962)]
[New Thread 0x7ff22e9ceb00 (LWP 21963)]
[Thread 0x7ff22e9ceb00 (LWP 21963) exited]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff1ddffb700 (LWP 21287)]
0x0000000000450da7 in update_secure_len (vec_idx=0x7ff1ddff27f8, secure_len=0x7ff1ddff27fc, id=13948, vq=0x7fe7992c8940)
at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:452
452 /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c: No such file or directory.
(gdb) bt
#0 0x0000000000450da7 in update_secure_len (vec_idx=0x7ff1ddff27f8, secure_len=0x7ff1ddff27fc, id=13948, vq=0x7fe7992c8940)
at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:452
#1 virtio_dev_merge_rx (count=1, pkts=0x7ff1ddffaa20, queue_id=0, dev=0x7fe79934df80) at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:516
#2 rte_vhost_enqueue_burst (dev=0x7fe79934df80, queue_id=0, pkts=0x7ff1ddffaa20, count=1) at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:561
#3 0x00000000007d343f in __netdev_dpdk_vhost_send (netdev=0x7fe799a5db80, qid=0, pkts=0x7ff1ddffaa20, cnt=1, may_steal=true) at lib/netdev-dpdk.c:1138
#4 0x00000000007d3b33 in netdev_dpdk_vhost_send (netdev=0x7fe799a5db80, qid=0, pkts=0x7ff1ddffaa20, cnt=1, may_steal=true) at lib/netdev-dpdk.c:1300
#5 0x00000000007167ef in netdev_send (netdev=0x7fe799a5db80, qid=0, buffers=0x7ff1ddffaa20, cnt=1, may_steal=true) at lib/netdev.c:742
#6 0x00000000006ea3f6 in dp_execute_cb (aux_=0x7ff1ddff3350, packets=0x7ff1ddffaa20, cnt=1, a=0x7ff1ddff3538, may_steal=true) at lib/dpif-netdev.c:3694
#7 0x0000000000720879 in odp_execute_actions (dp=0x7ff1ddff3350, packets=0x7ff1ddffaa20, cnt=1, steal=true, actions=0x7ff1ddff3538, actions_len=8,
dp_execute_action=0x6ea2f1 <dp_execute_cb>) at lib/odp-execute.c:535
#8 0x00000000006ea9c6 in dp_netdev_execute_actions (pmd=0x2474b10, packets=0x7ff1ddffaa20, cnt=1, may_steal=true, actions=0x7ff1ddff3538, actions_len=8) at lib/dpif-netdev.c:3841
#9 0x00000000006e9af4 in fast_path_processing (pmd=0x2474b10, packets=0x7ff1ddffa960, cnt=32, keys=0x7ff1ddff6940, batches=0x7ff1ddff4630, n_batches=0x7ff1ddffa888)
at lib/dpif-netdev.c:3499
#10 0x00000000006ea045 in dp_netdev_input__ (pmd=0x2474b10, packets=0x7ff1ddffa960, cnt=32, md_is_valid=false, port_no=11) at lib/dpif-netdev.c:3582
#11 0x00000000006ea105 in dp_netdev_input (pmd=0x2474b10, packets=0x7ff1ddffa960, cnt=32, port_no=11) at lib/dpif-netdev.c:3599
#12 0x00000000006e766b in dp_netdev_process_rxq_port (pmd=0x2474b10, port=0x1ff9f80, rxq=0x7fe798cab600) at lib/dpif-netdev.c:2602
#13 0x00000000006e7c24 in pmd_thread_main (f_=0x2474b10) at lib/dpif-netdev.c:2727
#14 0x00000000007723f2 in ovsthread_wrapper (aux_=0x20424b0) at lib/ovs-thread.c:340
#15 0x00007ff22db760a4 in start_thread (arg=0x7ff1ddffb700) at pthread_create.c:309
#16 0x00007ff22d3a287d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
(gdb) backtrace full
#0 0x0000000000450da7 in update_secure_len (vec_idx=0x7ff1ddff27f8, secure_len=0x7ff1ddff27fc, id=13948, vq=0x7fe7992c8940)
at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:452
wrapped_idx = 124
idx = 0
next_desc = 0 '\000'
len = 0
vec_id = 13015
#1 virtio_dev_merge_rx (count=1, pkts=0x7ff1ddffaa20, queue_id=0, dev=0x7fe79934df80) at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:516
secure_len = 0
vec_idx = 13015
pkt_len = 72
pkt_idx = 0
avail_idx = 0
res_base_idx = 933
res_cur_idx = 13948
success = 0 '\000'
vq = 0x7fe7992c8940
entry_success = 0
#2 rte_vhost_enqueue_burst (dev=0x7fe79934df80, queue_id=0, pkts=0x7ff1ddffaa20, count=1) at /home/stack/ovs-dpdk/dpdk-2.2.0/lib/librte_vhost/vhost_rxtx.c:561
No locals.
#3 0x00000000007d343f in __netdev_dpdk_vhost_send (netdev=0x7fe799a5db80, qid=0, pkts=0x7ff1ddffaa20, cnt=1, may_steal=true) at lib/netdev-dpdk.c:1138
vhost_qid = 0
tx_pkts = 4272456705
vhost_dev = 0x7fe799a5db80
virtio_dev = 0x7fe79934df80
cur_pkts = 0x7ff1ddffaa20
total_pkts = 1
start = 0
#4 0x00000000007d3b33 in netdev_dpdk_vhost_send (netdev=0x7fe799a5db80, qid=0, pkts=0x7ff1ddffaa20, cnt=1, may_steal=true) at lib/netdev-dpdk.c:1300
No locals.
#5 0x00000000007167ef in netdev_send (netdev=0x7fe799a5db80, qid=0, buffers=0x7ff1ddffaa20, cnt=1, may_steal=true) at lib/netdev.c:742
error = 0
#6 0x00000000006ea3f6 in dp_execute_cb (aux_=0x7ff1ddff3350, packets=0x7ff1ddffaa20, cnt=1, a=0x7ff1ddff3538, may_steal=true) at lib/dpif-netdev.c:3694
tx_qid = 0
aux = 0x7ff1ddff3350
depth = 0x7ff1ddffb550 [102/2145]
pmd = 0x2474b10
dp = 0x1fe9520
type = 1
p = 0x2008a00
i = 32753
#7 0x0000000000720879 in odp_execute_actions (dp=0x7ff1ddff3350, packets=0x7ff1ddffaa20, cnt=1, steal=true, actions=0x7ff1ddff3538, actions_len=8,
dp_execute_action=0x6ea2f1 <dp_execute_cb>) at lib/odp-execute.c:535
may_steal = true
type = 1
last_action = true
a = 0x7ff1ddff3538
left = 8
i = -570476240
#8 0x00000000006ea9c6 in dp_netdev_execute_actions (pmd=0x2474b10, packets=0x7ff1ddffaa20, cnt=1, may_steal=true, actions=0x7ff1ddff3538, actions_len=8) at lib/dpif-netdev.c:3841
aux = {pmd = 0x2474b10}
#9 0x00000000006e9af4 in fast_path_processing (pmd=0x2474b10, packets=0x7ff1ddffa960, cnt=32, keys=0x7ff1ddff6940, batches=0x7ff1ddff4630, n_batches=0x7ff1ddffa888)
at lib/dpif-netdev.c:3499
netdev_flow = 0x0
---Type <return> to continue, or q <return> to quit---
add_actions = 0x7ff1ddff34b0
match = {flow = {tunnel = {ip_dst = 0, ipv6_dst = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
ip_src = 0, ipv6_src = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, tun_id = 0, flags = 0,
ip_tos = 0 '\000', ip_ttl = 0 '\000', tp_src = 0, tp_dst = 0, gbp_id = 0, gbp_flags = 0 '\000', pad1 = "\000\000\000\000", metadata = {present = {map = 0, len = 0 '\000'},
tab = 0x0, opts = {u8 = '\000' <repeats 255 times>, gnv = {{opt_class = 0, type = 0 '\000', length = 0 '\000', r3 = 0 '\000', r2 = 0 '\000',
r1 = 0 '\000'} <repeats 63 times>}}}}, metadata = 0, regs = {0, 0, 0, 0, 0, 0, 0, 0}, skb_priority = 0, pkt_mark = 0, dp_hash = 0, in_port = {odp_port = 11,
ofp_port = 11}, recirc_id = 0, ct_state = 0, ct_zone = 0, ct_mark = 0, pad1 = "\000\000\000", ct_label = {u32 = {0, 0, 0, 0}, u64 = {lo = 0, hi = 0}}, conj_id = 0,
actset_output = 0, pad2 = "\000", dl_dst = {{ea = "\372\026>p-\235", be16 = {5882, 28734, 40237}}}, dl_src = {{ea = "\372\026>\346", <incomplete sequence \373>, be16 = {5882,
58942, 64284}}}, dl_type = 8, vlan_tci = 0, mpls_lse = {0, 0, 0, 0}, nw_src = 187869376, nw_dst = 171092160, ipv6_src = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ipv6_dst = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ipv6_label = 0, nw_frag = 0 '\000', nw_tos = 0 '\000',
nw_ttl = 128 '\200', nw_proto = 17 '\021', nd_target = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, arp_sha = {{ea = "\000\000\000\000\000", be16 = {0, 0, 0}}}, arp_tha = {{ea = "\000\000\000\000\000", be16 = {0, 0, 0}}}, tcp_flags = 0, pad3 = 0, tp_src = 0,
tp_dst = 0, igmp_group_ip4 = 0}, wc = {masks = {tunnel = {ip_dst = 0, ipv6_dst = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, ip_src = 0, ipv6_src = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0,
0, 0}}}, tun_id = 18446744073709551615, flags = 0, ip_tos = 0 '\000', ip_ttl = 0 '\000', tp_src = 0, tp_dst = 0, gbp_id = 0, gbp_flags = 0 '\000',
pad1 = "\000\000\000\000", metadata = {present = {map = 255, len = 255 '\377'}, tab = 0x0, opts = {u8 = '\000' <repeats 255 times>, gnv = {{opt_class = 0, type = 0 '\000',
length = 0 '\000', r3 = 0 '\000', r2 = 0 '\000', r1 = 0 '\000'} <repeats 63 times>}}}}, metadata = 0, regs = {0, 0, 0, 0, 0, 0, 0, 0}, skb_priority = 0,
pkt_mark = 0, dp_hash = 0, in_port = {odp_port = 4294967295, ofp_port = 65535}, recirc_id = 4294967295, ct_state = 0, ct_zone = 0, ct_mark = 0, pad1 = "\000\000\000",
ct_label = {u32 = {0, 0, 0, 0}, u64 = {lo = 0, hi = 0}}, conj_id = 0, actset_output = 0, pad2 = "\000", dl_dst = {{ea = "\377\377\377\377\377\377", be16 = {65535, 65535,
65535}}}, dl_src = {{ea = "\377\377\377\377\377\377", be16 = {65535, 65535, 65535}}}, dl_type = 65535, vlan_tci = 65535, mpls_lse = {0, 0, 0, 0}, nw_src = 0,
nw_dst = 0, ipv6_src = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ipv6_dst = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ipv6_label = 0, nw_frag = 3 '\003', nw_tos = 0 '\000',
nw_ttl = 0 '\000', nw_proto = 255 '\377', nd_target = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, arp_sha = {{ea = "\000\000\000\000\000", be16 = {0, 0, 0}}}, arp_tha = {{ea = "\000\000\000\000\000", be16 = {0, 0, 0}}}, tcp_flags = 0, pad3 = 0, tp_src = 1024,
tp_dst = 0, igmp_group_ip4 = 0}}, tun_md = {entry = {{loc = {len = -570475904, c = {next = 0x7fd26abab274, offset = 603983936, len = 32754}}, masked = 184}, {loc = {
len = 1790620276, c = {next = 0x6abab274, offset = -1735779648, len = 32743}}, masked = 128}, {loc = {len = 72, c = {next = 0x7fd26abab274, offset = 1790620276,
len = 32722}}, masked = 160}, {loc = {len = 1, c = {next = 0x0, offset = 54461248, len = 0}}, masked = 192}, {loc = {len = 0, c = {next = 0x80c00000001, offset = 63,
len = 4168693}}, masked = 120}, {loc = {len = -570475908, c = {next = 0x33f007ccfd0, offset = -1736326016, len = 32743}}, masked = 70}, {loc = {len = 54461248, c = {
next = 0x2000000020, offset = -1640693056, len = 32743}}, masked = 192}, {loc = {len = -1640693056, c = {next = 0x7fe79e3502c0, offset = -570474864, len = 32753}},
masked = 126}, {loc = {len = -570447792, c = {next = 0x7fe79e3502c0, offset = -570474720, len = 32753}}, masked = 64}, {loc = {len = -1662665728, c = {
next = 0x7ff1ddff4648, offset = 0, len = 0}}, masked = 192}, {loc = {len = 33529744, c = {next = 0x34aa970, offset = 55224688, len = 0}}, masked = 8}, {loc = {
len = 55224640, c = {next = 0x20006dce00, offset = 0, len = 32}}, masked = 192}, {loc = {len = -1728375104, c = {next = 0x7ff100000020, offset = 0, len = 0}},
masked = 72}, {loc = {len = -570474656, c = {next = 0x7d3b33 <netdev_dpdk_vhost_send+166>, offset = 33461616, len = 1}}, masked = 72}, {loc = {len = 7427118, c = {
next = 0x7ff1ddff3f60, offset = 7427177, len = 0}}, masked = 112}, {loc = {len = -570474592, c = {next = 0x716809 <netdev_send+110>, offset = -570474592, [48/2145]
len = 32513}}, masked = 12}, {loc = {len = -570474408, c = {next = 0x2, offset = -570474408, len = 32753}}, masked = 96}, {loc = {len = -570474528, c = {
next = 0x7fe79e32ea40, offset = -570474528, len = 32753}}, masked = 20}, {loc = {len = 4, c = {next = 0x7ff1ddff4030, offset = 1, len = 0}}, masked = true}, {loc = {
len = -570473568, c = {next = 0x6f40f6 <miniflow_extract+6199>, offset = -570473480, len = 32753}}, masked = 64}, {loc = {len = -570474448, c = {
next = 0x76ca73 <ofpbuf_prealloc_tailroom+29>, offset = 28, len = 0}}, masked = 144}, {loc = {len = -570453716, c = {next = 0x7ff1ddff9090, offset = 0,
len = -2067726336}}, masked = 16}, {loc = {len = -570473416, c = {next = 0x7ff1ddff4630, offset = 0, len = 11}}, masked = 26}, {loc = {len = -1640829662, c = {
next = 0x7185cf <nl_msg_put_uninit+66>, offset = 28, len = 0}}, masked = 144}, {loc = {len = -570453712, c = {next = 0x4, offset = 20, len = 0}}, masked = 144}, {
loc = {len = -570474304, c = {next = 0x7fe79e32ed22, offset = -570453872, len = 32753}}, masked = 8}, {loc = {len = -570474256, c = {
next = 0x76ca73 <ofpbuf_prealloc_tailroom+29>, offset = 4, len = 0}}, masked = 144}, {loc = {len = -570474192, c = {next = 0x7ff1ddff9090, offset = -570474208,
len = 32753}}, masked = 144}, {loc = {len = -570474208, c = {next = 0x76c2cd <ofpbuf_tailroom+40>, offset = -570453872, len = 32753}}, masked = 8}, {loc = {len = 1,
c = {next = 0x3a, offset = 3014684, len = 20}}, masked = 14}, {loc = {len = -570453716, c = {next = 0x7ff1ddff9090, offset = -570474112, len = 32753}}, masked = 147}, {
loc = {len = 28, c = {next = 0x2, offset = 54, len = 0}}, masked = false}, {loc = {len = -570474064, c = {next = 0x7185cf <nl_msg_put_uninit+66>, offset = 28, len = 0}},
masked = 144}, {loc = {len = -570453712, c = {next = 0x4, offset = 20, len = 0}}, masked = 144}, {loc = {len = -570474000, c = {next = 0x718d4f <nl_msg_end_nested+40>,
offset = -570473464, len = 32753}}, masked = 200}, {loc = {len = -570453712, c = {next = 0x7ff1ddff912c, offset = -570473904, len = 32753}}, masked = 225}, {loc = {
len = -570473904, c = {next = 0xffffffffffffffff, offset = -570473728, len = 32753}}, masked = 16}, {loc = {len = -570473728, c = {next = 0x7ff1ddff9090, offset = 20,
len = 0}}, masked = 12}, {loc = {len = -570473700, c = {next = 0x7ff1ddff9140, offset = -570473824, len = 0}}, masked = 2}, {loc = {len = 2, c = {next = 0x2,
offset = 0, len = 0}}, masked = true}, {loc = {len = 252645135, c = {next = 0x3333333333333333, offset = 1431655765, len = 1431655765}}, masked = 192}, {loc = {
len = 15, c = {next = 0x7ff1ddff42c0, offset = 7279429, len = 0}}, masked = 13}, {loc = {len = -570473768, c = {next = 0x6f138b <mhash_add+47>, offset = 0,
len = -695280421}}, masked = 240}, {loc = {len = 7279700, c = {next = 0x163a20f0000002a, offset = -570473616, len = 32753}}, masked = 37}, {loc = {len = 2, c = {
next = 0x7ff1ddff43f8, offset = 2, len = 0}}, masked = 192}, {loc = {len = -570473648, c = {next = 0x6e9230 <packet_batch_update+98>, offset = -570473632,
---Type <return> to continue, or q <return> to quit---
len = 32753}}, masked = 248}, {loc = {len = 0, c = {next = 0x2874be0, offset = 2, len = 0}}, masked = 248}, {loc = {len = 67576, c = {next = 0x107f8,
offset = 42421216, len = 0}}, masked = 224}, {loc = {len = -570473568, c = {next = 0x6e8cd0 <dp_netdev_count_packet+59>, offset = 0, len = 0}}, masked = 16}, {loc = {
len = -570472944, c = {next = 0x6e96f1 <emc_processing+753>, offset = -570452912, len = 32753}}, masked = 4}, {loc = {len = -570447736, c = {next = 0x7ff1ddff4630,
offset = -570463936, len = 32753}}, masked = 32}, {loc = {len = -570447520, c = {next = 0x2474b10, offset = -695280421, len = 0}}, masked = false}, {loc = {len = 16,
c = {next = 0xb00000000, offset = 1883117306, len = 385522989}}, masked = 62}, {loc = {len = 187869376, c = {next = 0x1180000000000000, offset = 0, len = 0}},
masked = 205}, {loc = {len = -570453072, c = {next = 0x8, offset = -570473344, len = 32753}}, masked = 115}, {loc = {len = 28, c = {next = 0x7ff1ddff93b0,
offset = -570452916, len = 32753}}, masked = 176}, {loc = {len = -570473296, c = {next = 0x76cc93 <ofpbuf_put_uninit+47>, offset = 28, len = 0}}, masked = 176}, {
loc = {len = -570453072, c = {next = 0x7ff1ddff9450, offset = -570473248, len = 32753}}, masked = 207}, {loc = {len = 28, c = {next = 0x7ff1ddff93b0, offset = -570452912,
len = 32753}}, masked = 4}, {loc = {len = 20, c = {next = 0x7ff1ddff93b0, offset = -570473184, len = 32753}}, masked = 79}, {loc = {len = 20, c = {
next = 0x7ff1ddff93b0, offset = -570452912, len = 32753}}, masked = 76}, {loc = {len = -570473088, c = {next = 0x7326e1 <commit_masked_set_action+211>,
offset = -570473088, len = 32753}}, masked = 80}, {loc = {len = 0, c = {next = 0x20, offset = 0, len = 0}}, masked = false}, {loc = {len = -570473072, c = {
next = 0x7ff22d3af4cd <__GI___clock_gettime+29>, offset = -570472884, len = 32753}}, masked = 221}}, alloc_offset = -570472960, valid = false}}
error = 28
actions_stub = {3467824696331874336, 34359803912, 3618677704915693616, 3684000802750543136, 2319969672871551029, 7003132829078926899, 3467830193890340896, 2320532622858723379,
3702275768603127856, 3467878585286537248, 2319688197878063156, 7076034831001334069, 3467824696332136480, 2319460599860568114, 7378620448010940720, 3467884301891560992,
2320532623663898672, 3832598695705076016, 3467884069963327008, 2319414420389109861, 3690454038725276982, 3900171385841137440, 2333768544622616627, 3546057160924358198,
3467880771428234784, 2334612968848105573, 7219868561263571767, 3611939897292698400, 2334045620797644900, 3545212946380502323, 3467827994866955040, 2320814098606989364,
4122799379513881904, 6998646812783751200, 2321993874654240825, 3832598884683638118, 3467827994867085344, 2320814098606989360, 3904656272563123504, 3467833492425158688,
2321658522849255480, 3832598682820174129, 729695728449970464, 55049728, 3724490448, 140676788341880, 140676788337872, 55047280, 6931125, 0, 140676788336384, 7052771, 55049664,
140676788336736, 140676788336464, 55047264, 18446744073709551615, 0, 140675748258320, 8251086490, 140675748267904, 422016453882914279, 55264098112, 140675748258320}
actions = {base = 0x7ff1ddff3530, data = 0x7ff1ddff3538, size = 8, allocated = 512, header = 0x0, msg = 0x0, list_node = {prev = 0xcccccccccccccccc, next = 0xcccccccccccccccc},
source = OFPBUF_STUB}
put_actions = {base = 0x7ff1ddff3730, data = 0x7ff1ddff3730, size = 0, allocated = 512, header = 0x0, msg = 0x0, list_node = {prev = 0xcccccccccccccccc,
next = 0xcccccccccccccccc}, source = OFPBUF_STUB}
ufid = {u32 = {3036086294, 2000505611, 3082407737, 2263132681}, u64 = {lo = 8592106177745584150, hi = 9720080854486208313}}
slow_stub = {47244640256, 42949672970, 140675748267904, 140675748267936, 15569323936, 140675748258320, 140675748267936, 140675748267904, 140675748267904, 140676788348712, 7199703,
140675748267904, 140675748267904, 16991402557314947260, 140675748258320, 140675748267840, 140675748267904, 140675748267968, 140676788336624, 7068963, 140676788336624, 33996736,
1, 33996736, 140676788336672, 7069473, 140675748267904, 140675748267904, 0, 140676788337552, 140676788340688, 7074309, 140676788342544, 140676788341872, 0, 0, 33461952,
33461952, 140676788341872, 140676788342544, 55222160, 15, 0 <repeats 22 times>}
PKT_ARRAY_SIZE = 32
rules = 0x7ff1ddff3370
dp = 0x1fe9520
flow_cache = 0x2474b88
miss_cnt = 25
lost_cnt = 0
any_miss = true
i = 24
#10 0x00000000006ea045 in dp_netdev_input__ (pmd=0x2474b10, packets=0x7ff1ddffa960, cnt=32, md_is_valid=false, port_no=11) at lib/dpif-netdev.c:3582
PKT_ARRAY_SIZE = 32
keys = 0x7ff1ddff6940
batches = 0x7ff1ddff4630
now = 11725351
newcnt = 32
n_batches = 0
i = 848966041873
#11 0x00000000006ea105 in dp_netdev_input (pmd=0x2474b10, packets=0x7ff1ddffa960, cnt=32, port_no=11) at lib/dpif-netdev.c:3599
No locals.
#12 0x00000000006e766b in dp_netdev_process_rxq_port (pmd=0x2474b10, port=0x1ff9f80, rxq=0x7fe798cab600) at lib/dpif-netdev.c:2602
packets = {0x7fe79e32a040, 0x7fe79e32a980, 0x7fe79e333440, 0x7fe79e38ce00, 0x7fe79e38d740, 0x7fe79e38e080, 0x7fe79e38e9c0, 0x7fe79e38f300, 0x7fe79cf6b780, 0x7fe79e3752c0,
0x7fe79bce1100, 0x7fe79cf6c0c0, 0x7fe79cf6ca00, 0x7fe79cf6d340, 0x7fe79e36c800, 0x7fe79e36bec0, 0x7fe79bce1a40, 0x7fe79bce2cc0, 0x7fe79cf6dc80, 0x7fe79cf6e5c0, 0x7fe79e362180,
0x7fe79bce3f40, 0x7fe79cf6ef00, 0x7fe79cf6f840, 0x7fe79e358d80, 0x7fe79bce51c0, 0x7fe79e341b80, 0x7fe79cf71d40, 0x7fe79e342e00, 0x7fe79e33ffc0, 0x7fe79e32f380, 0x7fe79e32ea40}
error = 0
cnt = 32
#13 0x00000000006e7c24 in pmd_thread_main (f_=0x2474b10) at lib/dpif-netdev.c:2727
---Type <return> to continue, or q <return> to quit---
pmd = 0x2474b10
lc = 817
poll_list = 0x7ff1a0006470
port_seq = 7
poll_cnt = 8
i = 6
#14 0x00000000007723f2 in ovsthread_wrapper (aux_=0x20424b0) at lib/ovs-thread.c:340
next_id = {count = 105}
auxp = 0x20424b0
aux = {start = 0x6e7a88 <pmd_thread_main>, arg = 0x2474b10, name = "pmd", '\000' <repeats 12 times>}
id = 101
subprogram_name = 0x7ff1a00008e0 ""
#15 0x00007ff22db760a4 in start_thread (arg=0x7ff1ddffb700) at pthread_create.c:309
__res = <optimized out>
pd = 0x7ff1ddffb700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140676788369152, 9038264028258769960, 0, 33993552, 16, 140676788369152, -9039315163138876376, -9040406352898024408},
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#16 0x00007ff22d3a287d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
(gdb) info registers
rax 0x0 0
rbx 0x7ff1ddffa648 140676788364872
rcx 0x32df0 208368
rdx 0x0 0
rsi 0x0 0
rdi 0x0 0
rbp 0x7ff1ddff2c10 0x7ff1ddff2c10
rsp 0x7ff1ddff0fa0 0x7ff1ddff0fa0
r8 0x0 0
r9 0x7ff1a00c6960 140675749013856
r10 0x4000 16384
r11 0x3202 12802
r12 0x7ff1ddff3480 140676788335744
r13 0x20 32
r14 0x0 0
r15 0x7ff1ddffb700 140676788369152
rip 0x450da7 0x450da7 <rte_vhost_enqueue_burst+692>
eflags 0x13246 [ PF ZF IF #12 #13 RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb) gcore
warning: target file /proc/21152/cmdline contained unexpected null characters
Couldn't get registers: No such process.
(gdb) c
Continuing.
^ permalink raw reply [flat|nested] 3+ messages in thread