From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by dpdk.org (Postfix) with ESMTP id E6AE63772 for ; Sat, 10 Dec 2016 05:27:54 +0100 (CET) Received: by mail-wm0-f52.google.com with SMTP id t79so4366431wmt.0 for ; Fri, 09 Dec 2016 20:27:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=11kqKew0YYmCfeXknLuOim783tbnPwfrXf2am03g0R0=; b=HYoZcNzi0iJS72EzINFaQhR94JROx0WGZqsER3G4NDgSZGKmQicbIaWsM1Gs/g/nJw BVLA3j0DfnQOBkwNZAonLVvzWKFpt0NwNjxNNKIN+XnZJEnmiHlKgi0hdPTLaGIiwjFW 9DO0GcrkGwIOuivtdElYeiy73jeTHSVYD48CbfNJTXL3b9Fkrp7jZui7kBmsuoILPKuU loHMQVdu6dJOYMeE6CuOes1TPHvlTL4gH4VqDrol2EqZFEuGWN+sWTIVv/5Pb0kTakoz kHdwaZJg+gtAJZomPOe8a+vvem/4FjITMS7fvn+IMQa8En5DsbBzA47b7daDAsXD079v MrLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=11kqKew0YYmCfeXknLuOim783tbnPwfrXf2am03g0R0=; b=gmCxBYFSim2d7YQ4Jntcda33SSnGNJ2zl/1PazY2bUhpkG3yVgio3uCFEVklWggi06 azxsqROkgXoDyZ76eCyuDrKLhjgovB76dPvErQcjmM3kDmI9hf+hwIt203FJ9bKxxd/p x456ZwacrG0oVFa+dUyZOZQ2hmPDEnHqVT4aLHPh5Gwa8Bvv2/pRsdGxBwxIAsTvPFEk ubTZG2WK6pWRoYlEng4ccT4giHJe3QsDCYKR19vcH01VymjWCfvdTEJyuqIwb3qZTBZ9 3OzoZN3uK5M4vIIAxccb29x///M7As/dGWvMCCzV5K9wdWgLtuHxtStTfEmVLPY8TBC+ chBw== X-Gm-Message-State: AKaTC012nw4Ffio8S9eKnHjwAMSBeeOqyxVEWnDJOYDyD5b5C757lz77lju2VXZ3o8qxUoopV2kWw9XLUqfCQA== X-Received: by 10.28.145.134 with SMTP id t128mr1014842wmd.50.1481344074249; Fri, 09 Dec 2016 20:27:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.80.146.164 with HTTP; Fri, 9 Dec 2016 20:27:53 -0800 (PST) In-Reply-To: References: From: Joo Kim Date: Fri, 9 Dec 2016 20:27:53 -0800 Message-ID: To: dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] After a port is bound to dpdk, when 'ip addr' or 'ip link add ~vlan' command against non-dpdk port , Linux kernel hangs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Dec 2016 04:27:55 -0000 This issue(https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195 )seems to be some similar to what I am facing. So, I tried its fix patches ( https://github.com/openstack/fuel-plugin-ovs/blob/master/ovs_build/dpdk_2.2.0/debian/patches/ubuntu-backport-36-pci-identify-devices-not-managed-by-any-kernel-drive.patch https://github.com/openstack/fuel-plugin-ovs/blob/master/ovs_build/dpdk_2.2.0/debian/patches/ubuntu-backport-37-pci-ignore-devices-already-managed-in-Linux-when-map.patch ) But, it didnot help. BTW, what I meant by 'Linux kernel hangs' is soft lockup. (I see following msg in dmesg [252460.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322] [252488.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322] [252493.698003] INFO: rcu_sched self-detected stall on CPU { 1} (t=60001 jiffies g=9321477 c=9321476 q=0) [252493.699003] INFO: rcu_sched detected stalls on CPUs/tasks: [252520.127002] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322] [252548.127003] BUG: soft lockup - CPU#1 stuck for 22s! [ifconfig:25322] ) On Fri, Dec 9, 2016 at 3:27 PM, Joo Kim wrote: > Hello, > I am using dpdk 2.2.0 in this VM where several NIC ports are available. > > Also, uio, kni drivers are installed. > [root@mylinux]# lsmod | grep uio > igb_uio 13224 0 > uio 19259 1 igb_uio > [root@mylinux]# lsmod | grep kni > rte_kni 294375 0 > i2c_core 40325 2 i2c_dev,rte_kni > [root@mylinux]# uname -r > 3.10.0-229.20.1.el7.x86_64 > > Weirdly, once I bind a network port to dpdk, after that, when I run 'ip > addr' or 'ip link add ~vlan' command against non-dpdk port , this VM's > Linux kernel hangs. > > For example, when I run 'ip link add ~vlan' command, then following > dmesg is seen and Linux hangs. > > Any idea on root cause/how to fix? > > {noformat} > [ 4884.158003] virtio dm_mirror dm_region_hash dm_log dm_mod loop [last > unloaded: openvswitch] > [ 4884.158003] CPU: 3 PID: 15060 Comm: dpdk_nic_bind.p Tainted: GF > O-------------- 3.10.0-229.20.1.el7.x86_64 #1 > [ 4884.158003] Hardware name: Bochs ovs-cloudinit-vm, BIOS Bochs 01/01/2011 > [ 4884.158003] task: ffff88038d045b00 ti: ffff880390e34000 task.ti: > ffff880390e34000 > [ 4884.158003] RIP: 0010:[] [] > virtnet_send_command.constprop.29+0x103/0x160 [virtio_net] > [ 4884.158003] RSP: 0018:ffff880390e379f8 EFLAGS: 00000246 > [ 4884.158003] RAX: 0000000000000000 RBX: 00000000000163e0 RCX: > ffff88038fbd3000 > [ 4884.158003] RDX: 000000000000c0f0 RSI: ffff880390e379fc RDI: > ffff88038e140000 > [ 4884.158003] RBP: ffff880390e37a90 R08: 00000000000163e0 R09: > ffffffffa005059c > [ 4884.158003] R10: 0000000000000003 R11: 0000000000000000 R12: > 0000000000000003 > [ 4884.158003] R13: 0000000000000000 R14: 0000000000000003 R15: > ffff880390e379e8 > [ 4884.158003] FS: 00007fc39b216740(0000) GS:ffff8803a80c0000(0000) knl > GS:0000000000000000 > [ 4884.158003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 4884.158003] CR2: 00007fc39b221000 CR3: 000000038cb6f000 CR4: > 00000000000007e0 > [ 4884.158003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 4884.158003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > [ 4884.158003] Stack: > [ 4884.158003] 000000000102ff46 ffff880390e37a20 ffff880390e37aa8 > ffff880390e37a40 > [ 4884.158003] ffffffff815bb3ae ffffea000e438dc2 00000002000009fa > 0000000000000000 > [ 4884.158003] 0000000000000000 ffffea000e438dc2 00000001000009f9 > 0000000000000000 > [ 4884.158003] Call Trace: > [ 4884.158003] [] ? fib6_clean_all+0xae/0x100 > [ 4884.158003] [] virtnet_vlan_rx_kill_vid+0x50/0x90 > [virtio_net] > [ 4884.158003] [] vlan_vid_del+0x113/0x170 > [ 4884.158003] [] vlan_device_event+0x65b/0x690 [8021q] > [ 4884.158003] [] notifier_call_chain+0x4c/0x70 > [ 4884.158003] [] raw_notifier_call_chain+0x16/0x20 > [ 4884.158003] [] call_netdevice_notifiers+0x2d/0x60 > [ 4884.158003] [] dev_close_many+0xcc/0x120 > [ 4884.158003] [] rollback_registered_many+0xa8/0x220 > [ 4884.158003] [] rollback_registered+0x40/0x70 > [ 4884.158003] [] unregister_netdevice_queue+0x48/0x80 > [ 4884.158003] [] unregister_netdev+0x1c/0x30 > [ 4884.158003] [] virtnet_remove+0x43/0x80 [virtio_net] > [ 4884.158003] [] virtio_dev_remove+0x49/0xd0 [virtio] > [ 4884.158003] [] __device_release_driver+0x7f/0xf0 > [ 4884.158003] [] device_release_driver+0x23/0x30 > [ 4884.158003] [] bus_remove_device+0x108/0x180 > [ 4884.158003] [] device_del+0x135/0x1f0 > [ 4884.158003] [] device_unregister+0x1e/0x60 > [ 4884.158003] [] unregister_virtio_device+0x16/0x30 > [virtio] > [ 4884.158003] [] virtio_pci_remove+0x2b/0x70 > [virtio_pci] > [ 4884.158003] [] pci_device_remove+0x3b/0xb0 > [ 4884.158003] [] __device_release_driver+0x7f/0xf0 > [ 4884.158003] [] device_release_driver+0x23/0x30 > [ 4884.158003] [] driver_unbind+0xbd/0xe0 > [ 4884.158003] [] drv_attr_store+0x24/0x40 > [ 4884.158003] [] sysfs_write_file+0xc6/0x140 > [ 4884.158003] [] vfs_write+0xbd/0x1e0 > [ 4884.158003] [] ? trace_do_page_fault+0x43/0x100 > [ 4884.158003] [] SyS_write+0x58/0xb0 > [ 4884.158003] [] system_call_fastpath+0x16/0x1b > [ 4884.158003] Code: c5 70 ff ff ff e8 2e 11 fe ff 48 8b 7b 08 e8 c5 02 fe > ff eb 12 0f 1f 00 48 8b 7b 08 e8 77 ff fd ff 84 c0 75 17 f3 90 48 8b 7b 08 > <48> 8d b5 6c ff ff ff e8 01 01 fe ff 48 85 c0 74 dc 80 bd 69 ff > {noformat} >