* [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs
@ 2016-12-09 23:27 Joo Kim
2016-12-10 4:27 ` Joo Kim
2017-01-12 12:16 ` Ferruh Yigit
0 siblings, 2 replies; 3+ messages in thread
From: Joo Kim @ 2016-12-09 23:27 UTC (permalink / raw)
To: dev
Hello,
I am using dpdk 2.2.0 in this VM where several NIC ports are available.
Also, uio, kni drivers are installed.
[root@mylinux]# lsmod | grep uio
igb_uio 13224 0
uio 19259 1 igb_uio
[root@mylinux]# lsmod | grep kni
rte_kni 294375 0
i2c_core 40325 2 i2c_dev,rte_kni
[root@mylinux]# uname -r
3.10.0-229.20.1.el7.x86_64
Weirdly, once I bind a network port to dpdk, after that, when I run 'ip
addr' or 'ip link add ~vlan' command against non-dpdk port , this VM's
Linux kernel hangs.
For example, when I run 'ip link add ~vlan' command, then following dmesg
is seen and Linux hangs.
Any idea on root cause/how to fix?
{noformat}
[ 4884.158003] virtio dm_mirror dm_region_hash dm_log dm_mod loop [last
unloaded: openvswitch]
[ 4884.158003] CPU: 3 PID: 15060 Comm: dpdk_nic_bind.p Tainted: GF
O-------------- 3.10.0-229.20.1.el7.x86_64 #1
[ 4884.158003] Hardware name: Bochs ovs-cloudinit-vm, BIOS Bochs 01/01/2011
[ 4884.158003] task: ffff88038d045b00 ti: ffff880390e34000 task.ti:
ffff880390e34000
[ 4884.158003] RIP: 0010:[<ffffffffa006f173>] [<ffffffffa006f173>]
virtnet_send_command.constprop.29+0x103/0x160 [virtio_net]
[ 4884.158003] RSP: 0018:ffff880390e379f8 EFLAGS: 00000246
[ 4884.158003] RAX: 0000000000000000 RBX: 00000000000163e0 RCX:
ffff88038fbd3000
[ 4884.158003] RDX: 000000000000c0f0 RSI: ffff880390e379fc RDI:
ffff88038e140000
[ 4884.158003] RBP: ffff880390e37a90 R08: 00000000000163e0 R09:
ffffffffa005059c
[ 4884.158003] R10: 0000000000000003 R11: 0000000000000000 R12:
0000000000000003
[ 4884.158003] R13: 0000000000000000 R14: 0000000000000003 R15:
ffff880390e379e8
[ 4884.158003] FS: 00007fc39b216740(0000) GS:ffff8803a80c0000(0000) knl
GS:0000000000000000
[ 4884.158003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4884.158003] CR2: 00007fc39b221000 CR3: 000000038cb6f000 CR4:
00000000000007e0
[ 4884.158003] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 4884.158003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 4884.158003] Stack:
[ 4884.158003] 000000000102ff46 ffff880390e37a20 ffff880390e37aa8
ffff880390e37a40
[ 4884.158003] ffffffff815bb3ae ffffea000e438dc2 00000002000009fa
0000000000000000
[ 4884.158003] 0000000000000000 ffffea000e438dc2 00000001000009f9
0000000000000000
[ 4884.158003] Call Trace:
[ 4884.158003] [<ffffffff815bb3ae>] ? fib6_clean_all+0xae/0x100
[ 4884.158003] [<ffffffffa006f3f0>] virtnet_vlan_rx_kill_vid+0x50/0x90
[virtio_net]
[ 4884.158003] [<ffffffff815e67e3>] vlan_vid_del+0x113/0x170
[ 4884.158003] [<ffffffffa02b681b>] vlan_device_event+0x65b/0x690 [8021q]
[ 4884.158003] [<ffffffff8160fdec>] notifier_call_chain+0x4c/0x70
[ 4884.158003] [<ffffffff8109d456>] raw_notifier_call_chain+0x16/0x20
[ 4884.158003] [<ffffffff814f78fd>] call_netdevice_notifiers+0x2d/0x60
[ 4884.158003] [<ffffffff814f7bac>] dev_close_many+0xcc/0x120
[ 4884.158003] [<ffffffff814f8ee8>] rollback_registered_many+0xa8/0x220
[ 4884.158003] [<ffffffff814f90a0>] rollback_registered+0x40/0x70
[ 4884.158003] [<ffffffff814fa388>] unregister_netdevice_queue+0x48/0x80
[ 4884.158003] [<ffffffff814fa3dc>] unregister_netdev+0x1c/0x30
[ 4884.158003] [<ffffffffa006e0e3>] virtnet_remove+0x43/0x80 [virtio_net]
[ 4884.158003] [<ffffffffa002f119>] virtio_dev_remove+0x49/0xd0 [virtio]
[ 4884.158003] [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
[ 4884.158003] [<ffffffff813d2d53>] device_release_driver+0x23/0x30
[ 4884.158003] [<ffffffff813d24c8>] bus_remove_device+0x108/0x180
[ 4884.158003] [<ffffffff813ce9e5>] device_del+0x135/0x1f0
[ 4884.158003] [<ffffffff813ceabe>] device_unregister+0x1e/0x60
[ 4884.158003] [<ffffffffa002f426>] unregister_virtio_device+0x16/0x30
[virtio]
[ 4884.158003] [<ffffffffa01c852b>] virtio_pci_remove+0x2b/0x70
[virtio_pci]
[ 4884.158003] [<ffffffff813097bb>] pci_device_remove+0x3b/0xb0
[ 4884.158003] [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
[ 4884.158003] [<ffffffff813d2d53>] device_release_driver+0x23/0x30
[ 4884.158003] [<ffffffff813d16bd>] driver_unbind+0xbd/0xe0
[ 4884.158003] [<ffffffff813d0bf4>] drv_attr_store+0x24/0x40
[ 4884.158003] [<ffffffff8123cfc6>] sysfs_write_file+0xc6/0x140
[ 4884.158003] [<ffffffff811c6a2d>] vfs_write+0xbd/0x1e0
[ 4884.158003] [<ffffffff8160fce3>] ? trace_do_page_fault+0x43/0x100
[ 4884.158003] [<ffffffff811c7478>] SyS_write+0x58/0xb0
[ 4884.158003] [<ffffffff81614409>] system_call_fastpath+0x16/0x1b
[ 4884.158003] Code: c5 70 ff ff ff e8 2e 11 fe ff 48 8b 7b 08 e8 c5 02 fe
ff eb 12 0f 1f 00 48 8b 7b 08 e8 77 ff fd ff 84 c0 75 17 f3 90 48 8b 7b 08
<48> 8d b5 6c ff ff ff e8 01 01 fe ff 48 85 c0 74 dc 80 bd 69 ff
{noformat}
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs
2016-12-09 23:27 [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs Joo Kim
@ 2016-12-10 4:27 ` Joo Kim
2017-01-12 12:16 ` Ferruh Yigit
1 sibling, 0 replies; 3+ messages in thread
From: Joo Kim @ 2016-12-10 4:27 UTC (permalink / raw)
To: dev
This issue(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195
)seems to be some similar to what I am facing.
So, I tried its fix patches
(
https://github.com/openstack/fuel-plugin-ovs/blob/master/ovs_build/dpdk_2.2.0/debian/patches/ubuntu-backport-36-pci-identify-devices-not-managed-by-any-kernel-drive.patch
https://github.com/openstack/fuel-plugin-ovs/blob/master/ovs_build/dpdk_2.2.0/debian/patches/ubuntu-backport-37-pci-ignore-devices-already-managed-in-Linux-when-map.patch
)
But, it didnot help.
BTW, what I meant by 'Linux kernel hangs' is soft lockup.
(I see following msg in dmesg
[252460.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
[252488.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
[252493.698003] INFO: rcu_sched self-detected stall on CPU { 1} (t=60001
jiffies g=9321477 c=9321476 q=0)
[252493.699003] INFO: rcu_sched detected stalls on CPUs/tasks:
[252520.127002] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
[252548.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322]
)
On Fri, Dec 9, 2016 at 3:27 PM, Joo Kim <itsolution@gmail.com> wrote:
> Hello,
> I am using dpdk 2.2.0 in this VM where several NIC ports are available.
>
> Also, uio, kni drivers are installed.
> [root@mylinux]# lsmod | grep uio
> igb_uio 13224 0
> uio 19259 1 igb_uio
> [root@mylinux]# lsmod | grep kni
> rte_kni 294375 0
> i2c_core 40325 2 i2c_dev,rte_kni
> [root@mylinux]# uname -r
> 3.10.0-229.20.1.el7.x86_64
>
> Weirdly, once I bind a network port to dpdk, after that, when I run 'ip
> addr' or 'ip link add ~vlan' command against non-dpdk port , this VM's
> Linux kernel hangs.
>
> For example, when I run 'ip link add ~vlan' command, then following
> dmesg is seen and Linux hangs.
>
> Any idea on root cause/how to fix?
>
> {noformat}
> [ 4884.158003] virtio dm_mirror dm_region_hash dm_log dm_mod loop [last
> unloaded: openvswitch]
> [ 4884.158003] CPU: 3 PID: 15060 Comm: dpdk_nic_bind.p Tainted: GF
> O-------------- 3.10.0-229.20.1.el7.x86_64 #1
> [ 4884.158003] Hardware name: Bochs ovs-cloudinit-vm, BIOS Bochs 01/01/2011
> [ 4884.158003] task: ffff88038d045b00 ti: ffff880390e34000 task.ti:
> ffff880390e34000
> [ 4884.158003] RIP: 0010:[<ffffffffa006f173>] [<ffffffffa006f173>]
> virtnet_send_command.constprop.29+0x103/0x160 [virtio_net]
> [ 4884.158003] RSP: 0018:ffff880390e379f8 EFLAGS: 00000246
> [ 4884.158003] RAX: 0000000000000000 RBX: 00000000000163e0 RCX:
> ffff88038fbd3000
> [ 4884.158003] RDX: 000000000000c0f0 RSI: ffff880390e379fc RDI:
> ffff88038e140000
> [ 4884.158003] RBP: ffff880390e37a90 R08: 00000000000163e0 R09:
> ffffffffa005059c
> [ 4884.158003] R10: 0000000000000003 R11: 0000000000000000 R12:
> 0000000000000003
> [ 4884.158003] R13: 0000000000000000 R14: 0000000000000003 R15:
> ffff880390e379e8
> [ 4884.158003] FS: 00007fc39b216740(0000) GS:ffff8803a80c0000(0000) knl
> GS:0000000000000000
> [ 4884.158003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 4884.158003] CR2: 00007fc39b221000 CR3: 000000038cb6f000 CR4:
> 00000000000007e0
> [ 4884.158003] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 4884.158003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 4884.158003] Stack:
> [ 4884.158003] 000000000102ff46 ffff880390e37a20 ffff880390e37aa8
> ffff880390e37a40
> [ 4884.158003] ffffffff815bb3ae ffffea000e438dc2 00000002000009fa
> 0000000000000000
> [ 4884.158003] 0000000000000000 ffffea000e438dc2 00000001000009f9
> 0000000000000000
> [ 4884.158003] Call Trace:
> [ 4884.158003] [<ffffffff815bb3ae>] ? fib6_clean_all+0xae/0x100
> [ 4884.158003] [<ffffffffa006f3f0>] virtnet_vlan_rx_kill_vid+0x50/0x90
> [virtio_net]
> [ 4884.158003] [<ffffffff815e67e3>] vlan_vid_del+0x113/0x170
> [ 4884.158003] [<ffffffffa02b681b>] vlan_device_event+0x65b/0x690 [8021q]
> [ 4884.158003] [<ffffffff8160fdec>] notifier_call_chain+0x4c/0x70
> [ 4884.158003] [<ffffffff8109d456>] raw_notifier_call_chain+0x16/0x20
> [ 4884.158003] [<ffffffff814f78fd>] call_netdevice_notifiers+0x2d/0x60
> [ 4884.158003] [<ffffffff814f7bac>] dev_close_many+0xcc/0x120
> [ 4884.158003] [<ffffffff814f8ee8>] rollback_registered_many+0xa8/0x220
> [ 4884.158003] [<ffffffff814f90a0>] rollback_registered+0x40/0x70
> [ 4884.158003] [<ffffffff814fa388>] unregister_netdevice_queue+0x48/0x80
> [ 4884.158003] [<ffffffff814fa3dc>] unregister_netdev+0x1c/0x30
> [ 4884.158003] [<ffffffffa006e0e3>] virtnet_remove+0x43/0x80 [virtio_net]
> [ 4884.158003] [<ffffffffa002f119>] virtio_dev_remove+0x49/0xd0 [virtio]
> [ 4884.158003] [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
> [ 4884.158003] [<ffffffff813d2d53>] device_release_driver+0x23/0x30
> [ 4884.158003] [<ffffffff813d24c8>] bus_remove_device+0x108/0x180
> [ 4884.158003] [<ffffffff813ce9e5>] device_del+0x135/0x1f0
> [ 4884.158003] [<ffffffff813ceabe>] device_unregister+0x1e/0x60
> [ 4884.158003] [<ffffffffa002f426>] unregister_virtio_device+0x16/0x30
> [virtio]
> [ 4884.158003] [<ffffffffa01c852b>] virtio_pci_remove+0x2b/0x70
> [virtio_pci]
> [ 4884.158003] [<ffffffff813097bb>] pci_device_remove+0x3b/0xb0
> [ 4884.158003] [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
> [ 4884.158003] [<ffffffff813d2d53>] device_release_driver+0x23/0x30
> [ 4884.158003] [<ffffffff813d16bd>] driver_unbind+0xbd/0xe0
> [ 4884.158003] [<ffffffff813d0bf4>] drv_attr_store+0x24/0x40
> [ 4884.158003] [<ffffffff8123cfc6>] sysfs_write_file+0xc6/0x140
> [ 4884.158003] [<ffffffff811c6a2d>] vfs_write+0xbd/0x1e0
> [ 4884.158003] [<ffffffff8160fce3>] ? trace_do_page_fault+0x43/0x100
> [ 4884.158003] [<ffffffff811c7478>] SyS_write+0x58/0xb0
> [ 4884.158003] [<ffffffff81614409>] system_call_fastpath+0x16/0x1b
> [ 4884.158003] Code: c5 70 ff ff ff e8 2e 11 fe ff 48 8b 7b 08 e8 c5 02 fe
> ff eb 12 0f 1f 00 48 8b 7b 08 e8 77 ff fd ff 84 c0 75 17 f3 90 48 8b 7b 08
> <48> 8d b5 6c ff ff ff e8 01 01 fe ff 48 85 c0 74 dc 80 bd 69 ff
> {noformat}
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs
2016-12-09 23:27 [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs Joo Kim
2016-12-10 4:27 ` Joo Kim
@ 2017-01-12 12:16 ` Ferruh Yigit
1 sibling, 0 replies; 3+ messages in thread
From: Ferruh Yigit @ 2017-01-12 12:16 UTC (permalink / raw)
To: Joo Kim, dev
On 12/9/2016 11:27 PM, Joo Kim wrote:
> Hello,
> I am using dpdk 2.2.0 in this VM where several NIC ports are available.
>
> Also, uio, kni drivers are installed.
> [root@mylinux]# lsmod | grep uio
> igb_uio 13224 0
> uio 19259 1 igb_uio
> [root@mylinux]# lsmod | grep kni
> rte_kni 294375 0
> i2c_core 40325 2 i2c_dev,rte_kni
Why kni module depends on i2c?
> [root@mylinux]# uname -r
> 3.10.0-229.20.1.el7.x86_64
>
> Weirdly, once I bind a network port to dpdk, after that, when I run 'ip
> addr' or 'ip link add ~vlan' command against non-dpdk port , this VM's
> Linux kernel hangs.
Is KNI module used at all in this point?
>
> For example, when I run 'ip link add ~vlan' command, then following dmesg
> is seen and Linux hangs.
>
> Any idea on root cause/how to fix?
>
<...>
> [ 4884.158003] Call Trace:
> [ 4884.158003] [<ffffffff815bb3ae>] ? fib6_clean_all+0xae/0x100
> [ 4884.158003] [<ffffffffa006f3f0>] virtnet_vlan_rx_kill_vid+0x50/0x90
> [virtio_net]
> [ 4884.158003] [<ffffffff815e67e3>] vlan_vid_del+0x113/0x170
> [ 4884.158003] [<ffffffffa02b681b>] vlan_device_event+0x65b/0x690 [8021q]
> [ 4884.158003] [<ffffffff8160fdec>] notifier_call_chain+0x4c/0x70
> [ 4884.158003] [<ffffffff8109d456>] raw_notifier_call_chain+0x16/0x20
> [ 4884.158003] [<ffffffff814f78fd>] call_netdevice_notifiers+0x2d/0x60
> [ 4884.158003] [<ffffffff814f7bac>] dev_close_many+0xcc/0x120
> [ 4884.158003] [<ffffffff814f8ee8>] rollback_registered_many+0xa8/0x220
> [ 4884.158003] [<ffffffff814f90a0>] rollback_registered+0x40/0x70
> [ 4884.158003] [<ffffffff814fa388>] unregister_netdevice_queue+0x48/0x80
> [ 4884.158003] [<ffffffff814fa3dc>] unregister_netdev+0x1c/0x30
> [ 4884.158003] [<ffffffffa006e0e3>] virtnet_remove+0x43/0x80 [virtio_net]
> [ 4884.158003] [<ffffffffa002f119>] virtio_dev_remove+0x49/0xd0 [virtio]
This looks like crash occurred when virtio unbind.
Are you using custom modules?
> [ 4884.158003] [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
> [ 4884.158003] [<ffffffff813d2d53>] device_release_driver+0x23/0x30
> [ 4884.158003] [<ffffffff813d24c8>] bus_remove_device+0x108/0x180
> [ 4884.158003] [<ffffffff813ce9e5>] device_del+0x135/0x1f0
> [ 4884.158003] [<ffffffff813ceabe>] device_unregister+0x1e/0x60
> [ 4884.158003] [<ffffffffa002f426>] unregister_virtio_device+0x16/0x30
> [virtio]
> [ 4884.158003] [<ffffffffa01c852b>] virtio_pci_remove+0x2b/0x70
> [virtio_pci]
> [ 4884.158003] [<ffffffff813097bb>] pci_device_remove+0x3b/0xb0
> [ 4884.158003] [<ffffffff813d2cbf>] __device_release_driver+0x7f/0xf0
> [ 4884.158003] [<ffffffff813d2d53>] device_release_driver+0x23/0x30
> [ 4884.158003] [<ffffffff813d16bd>] driver_unbind+0xbd/0xe0
> [ 4884.158003] [<ffffffff813d0bf4>] drv_attr_store+0x24/0x40
> [ 4884.158003] [<ffffffff8123cfc6>] sysfs_write_file+0xc6/0x140
> [ 4884.158003] [<ffffffff811c6a2d>] vfs_write+0xbd/0x1e0
> [ 4884.158003] [<ffffffff8160fce3>] ? trace_do_page_fault+0x43/0x100
> [ 4884.158003] [<ffffffff811c7478>] SyS_write+0x58/0xb0
> [ 4884.158003] [<ffffffff81614409>] system_call_fastpath+0x16/0x1b
> [ 4884.158003] Code: c5 70 ff ff ff e8 2e 11 fe ff 48 8b 7b 08 e8 c5 02 fe
> ff eb 12 0f 1f 00 48 8b 7b 08 e8 77 ff fd ff 84 c0 75 17 f3 90 48 8b 7b 08
> <48> 8d b5 6c ff ff ff e8 01 01 fe ff 48 85 c0 74 dc 80 bd 69 ff
> {noformat}
>
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2017-01-12 12:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-09 23:27 [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs Joo Kim
2016-12-10 4:27 ` Joo Kim
2017-01-12 12:16 ` Ferruh Yigit
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).