From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qg0-f47.google.com (mail-qg0-f47.google.com [209.85.192.47]) by dpdk.org (Postfix) with ESMTP id 9551A2C47 for ; Mon, 21 Mar 2016 09:17:10 +0100 (CET) Received: by mail-qg0-f47.google.com with SMTP id y89so147371512qge.2 for ; Mon, 21 Mar 2016 01:17:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=btWeRKn5phvvf7+fcu+UaI/fhRMzQ1e0t0NWHJsl0ao=; b=H9eWxeMlQvi8GYq72YwpGuOkS+e+55R/JlJiuz1k23B7Jnu8vyTrUImzvON/kg7rw4 fyYpib6SwlHTYQX43ONr14BtXGhTJbjwg6QY+74e11gHZeBImaJvpJSZrFKQvAPkfgC3 hSSrJUsyKbUlhwN082TiWJOiVYVQvbW+cb1wOO6CkBhf4FsylUwShgCg21qP82PjDu60 /aMRESQEBOVJGcRXSFGNgJYllHVv8I2tHOh/6UOo31i0aZXpcUd8601ihXLxQXMPSyGD q/RH6D29FAKbya7FDld5Q9SObl1XCLXhAJTNP7Gvp6E7RCjUkf5Z1kEa4JZsQpMzrE4s 2dPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=btWeRKn5phvvf7+fcu+UaI/fhRMzQ1e0t0NWHJsl0ao=; b=LIcG62OItqh9jp/NJUWb+d3S6kelGqi5eC0XSwt+WxzcTcGcsjbj4HuisdCP9nDnvR E5+MMw0TP5rcOcQPgKDHwoPSsGcK0hYiiIUdAwYx4jzqMtQU4WFbynVP9VQAf+p1H8FM W5hJ7VhLGN8EJq+tCBkqPZ0M3iycmBXeXJjyL3lhnkz/eDxUqTsJolM5Y2bu/agnwWaU MX40HIgUSTyRsP4K3OquTG7UtPX1FJ3ro+FSdNZqdBCIfI5XK00fxygk3Kj7BZ0VjJZ2 n0OwPPB7trgUyAKEb/XRobJnIIsGfPu8i9dcnKwLixb3iMyfYzUhYWrhog0kYQu2lQKZ BHYA== X-Gm-Message-State: AD7BkJLxtWNrMSsCiD1lHuMB8hrLJrgJoKnJJVAOr0HPjmCvVVo6In3ufA0ChiKtiieKir1LjZDYyx8wRz3SVknb X-Received: by 10.140.172.212 with SMTP id s203mr39641227qhs.41.1458548230007; Mon, 21 Mar 2016 01:17:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.169.6 with HTTP; Mon, 21 Mar 2016 01:16:50 -0700 (PDT) In-Reply-To: References: From: Christian Ehrhardt Date: Mon, 21 Mar 2016 09:16:50 +0100 Message-ID: To: dev , discuss@openvswitch.org, diproiettod@vmware.com, ciara.loftus@intel.com, Thiago Martins Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Issues with openvswitch2.5+dpdk2.2+kvm/virtio-pci X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Mar 2016 08:17:11 -0000 FYI - a user also reported that the same issue also affects non-virtual environments when using the BNX2X_PMD Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd On Fri, Mar 18, 2016 at 4:20 PM, Christian Ehrhardt < christian.ehrhardt@canonical.com> wrote: > Hi, > I was trying to replicate a setup that I have working on physical devices > (ixgbe) under kvm since there is a virtio pmd driver. > > TL;DR: > - under KVM with virtio-pci (working on baremetal with ixgbe cards) > - adding dpdk port to ovs fails with memzone already exists > and causes a segfault > - I couldn't find a solution in similar mails that popped up here > recently, any help or pointer appreciated. > > ## Details ## > I thought I've read that others have it working I thought that would be a > great way to gain more debug control of the environment, but something > seems to be eluding me. > > There were quite some similar mails on the List recently, but none seemed > to hit the same issue as I do. At least none of the tunings/workarounds > seemed to apply to me. > As versions I have Openvswitch 2.5, DPDK 2.2, Qemu 2.5, Kernel 4.4 - so a > fairly recent software stack. > > The super-short repro summary is: > 1. starting ovs-dpdk like > ovs-vswitchd --dpdk -c 0x1 -n 4 --pci-blacklist 0000:00:03.0 -m 2048 -- > unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info > --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log > --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor > 2. add a bridge and a ovs dpdk port > ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk > ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk > > The log of the initialization after #1 looks good to me - I can see two of > my three virtio devices recognized and one blacklisted. > Memory allocation looks good, ... I'll attach the log at the end of the > mail > > > ## ISSUE ## > But when I add a port and refer to one of the dpdk ports it fails with the > following: > ovs-vsctl[14023]: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port > ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk > ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone > already exists > ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone > already exists > ovs-vswitchd[13903]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone > already exists > kernel: show_signal_msg: 18 callbacks suppressed > kernel: pmd12[14025]: segfault at 2 ip 00007f3eb205eab2 sp > 00007f3e3dffa590 error 4 in libdpdk.so.0[7f3eb1fdf000+1e9000] > ovs-vswitchd[13902]: ovs|00003|daemon_unix(monitor)|ERR|1 crashes: pid > 13903 died, killed (Segmentation fault), core dumped, restarting > systemd-udevd[14040]: Could not generate persistent MAC address for > ovs-netdev: No such file or directory > kernel: device ovs-netdev entered promiscuous mode > ovs-vswitchd[14036]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone > already exists > ovs-vswitchd[14036]: RING: Cannot reserve memory > kernel: device ovsdpdkbr0 entered promiscuous mode > ovs-vswitchd[14036]: EAL: memzone_reserve_aligned_thread_unsafe(): memzone > already exists > ovs-vswitchd[14036]: RING: Cannot reserve memory > > > ## Experiments (failed) ## > I thought it could be related to all the multiqueue chances that recently > going in. > My usual setup has 4 vCPUs and 4 queues per virtio-net device. > I tried them with only 1 of 4 queues, also with only 1 queue defined and > only 1 CPU - but all fail the same way. > I have testpmd and l2fwd on the same devices working, so I hope they are > not totally set up badly. > > I also tried hilarious things like reassigning to uio_pci_generic before, > but well its virtio_pmd eventually anyways - so it made no difference. > > From how it appears I felt that it could be related to the old discussions > around > [1] http://dpdk.org/ml/archives/dev/2015-May/017589.html > [2] http://openvswitch.org/pipermail/dev/2015-March/052344.html > But they are (partially) applied upstream already and the issue doesn't > 100% match the old discussions. > > > ## Logs ## > [3] log of openvswitch start: > systemd[1]: Starting Open vSwitch Internal Unit... > ovs-ctl[13868]: * Starting ovsdb-server > ovs-vsctl[13893]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- > init -- set Open_vSwitch . db-version=7.12.1 > ovs-vsctl[13898]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set > Open_vSwitch . ovs-version=2.5.0 > "external-ids:system-id=\"8ddb892c-53a5-410d-a765-0031ad6eb1be\"" > "system-type=\"Ubuntu\"" "system-version=\"16.04-xenial\"" > ovs-ctl[13868]: * Configuring Open vSwitch system IDs > ovs-ctl[13868]: 2016-03-18T14:28:28Z|00001|dpdk|INFO|No -vhost_sock_dir > provided - defaulting to /var/run/openvswitch > ovs-vswitchd[13900]: ovs|00001|dpdk|INFO|No -vhost_sock_dir provided - > defaulting to /var/run/openvswitch > ovs-ctl[13868]: EAL: Detected lcore 0 as core 0 on socket 0 > ovs-ctl[13868]: EAL: Detected lcore 1 as core 0 on socket 0 > ovs-ctl[13868]: EAL: Detected lcore 2 as core 0 on socket 0 > ovs-ctl[13868]: EAL: Detected lcore 3 as core 0 on socket 0 > ovs-ctl[13868]: EAL: Support maximum 128 logical core(s) by configuration. > ovs-ctl[13868]: EAL: Detected 4 lcore(s) > ovs-ctl[13868]: EAL: No free hugepages reported in hugepages-1048576kB > ovs-ctl[13868]: EAL: VFIO modules not all loaded, skip VFIO support... > ovs-ctl[13868]: EAL: Setting up physically contiguous memory... > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3eaf800000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x5ac00000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e54a00000 (size = > 0x5ac00000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e53c00000 (size = 0xc00000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x1200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e52800000 (size = > 0x1200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e52200000 (size = 0x400000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51e00000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51800000 (size = 0x400000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e51400000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e50600000 (size = 0xc00000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x1000000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4f400000 (size = > 0x1000000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4f000000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4ec00000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4e800000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4e400000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4de00000 (size = 0x400000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3e4da00000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x5ac00000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df2c00000 (size = > 0x5ac00000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0xc00000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df1e00000 (size = 0xc00000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df1800000 (size = 0x400000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x1000000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df0600000 (size = > 0x1000000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3df0200000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3defc00000 (size = 0x400000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3def800000 (size = 0x200000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x400000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3def200000 (size = 0x400000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x600000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3deea00000 (size = 0x600000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x1800000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3ded000000 (size = > 0x1800000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x600000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3dec800000 (size = 0x600000) > ovs-ctl[13868]: EAL: Ask a virtual area of 0x200000 bytes > ovs-ctl[13868]: EAL: Virtual area found at 0x7f3dec400000 (size = 0x200000) > ovs-ctl[13868]: EAL: Requesting 1024 pages of size 2MB from socket 0 > ovs-ctl[13868]: EAL: TSC frequency is ~2397235 KHz > ovs-vswitchd[13900]: EAL: TSC frequency is ~2397235 KHz > ovs-ctl[13868]: EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> > using unreliable clock cycles ! > ovs-ctl[13868]: EAL: Master lcore 0 is ready (tid=b2f7ab00;cpuset=[0]) > ovs-ctl[13868]: EAL: PCI device 0000:00:03.0 on NUMA socket -1 > ovs-ctl[13868]: EAL: probe driver: 1af4:1000 rte_virtio_pmd > ovs-ctl[13868]: EAL: Device is blacklisted, not initializing > ovs-ctl[13868]: EAL: PCI device 0000:00:04.0 on NUMA socket -1 > ovs-ctl[13868]: EAL: probe driver: 1af4:1000 rte_virtio_pmd > ovs-vswitchd[13900]: EAL: WARNING: cpu flags constant_tsc=yes > nonstop_tsc=no -> using unreliable clock cycles ! > ovs-ctl[13868]: EAL: PCI device 0000:00:05.0 on NUMA socket -1 > ovs-ctl[13868]: EAL: probe driver: 1af4:1000 rte_virtio_pmd > ovs-vswitchd[13900]: EAL: Master lcore 0 is ready (tid=b2f7ab00;cpuset=[0]) > ovs-vswitchd[13900]: EAL: PCI device 0000:00:03.0 on NUMA socket -1 > ovs-vswitchd[13900]: EAL: probe driver: 1af4:1000 rte_virtio_pmd > ovs-vswitchd[13900]: EAL: Device is blacklisted, not initializing > ovs-vswitchd[13900]: EAL: PCI device 0000:00:04.0 on NUMA socket -1 > ovs-vswitchd[13900]: EAL: probe driver: 1af4:1000 rte_virtio_pmd > ovs-vswitchd[13900]: EAL: PCI device 0000:00:05.0 on NUMA socket -1 > ovs-vswitchd[13900]: EAL: probe driver: 1af4:1000 rte_virtio_pmd > ovs-ctl[13868]: Zone 0: name:, phys:0x1495fdec0, > len:0x2080, virt:0x7f3e4dbfdec0, socket_id:0, flags:0 > ovs-ctl[13868]: Zone 1: name:, phys:0xb7d75f00, > len:0x28a0c0, virt:0x7f3e4df75f00, socket_id:0, flags:0 > ovs-ctl[13868]: Zone 2: name:, phys:0xb7d467c0, > len:0x2f700, virt:0x7f3e4df467c0, socket_id:0, flags:0 > ovs-ctl[13868]: Zone 3: name:, phys:0xb7d43000, len:0x2000, > virt:0x7f3e4df43000, socket_id:0, flags:0 > ovs-ctl[13868]: Zone 4: name:, phys:0xb7d41fc0, > len:0x1000, virt:0x7f3e4df41fc0, socket_id:0, flags:0 > ovs-ctl[13868]: Zone 5: name:, phys:0xb7d3f000, len:0x2000, > virt:0x7f3e4df3f000, socket_id:0, flags:0 > ovs-ctl[13868]: Zone 6: name:, phys:0xb7d3dfc0, > len:0x1000, virt:0x7f3e4df3dfc0, socket_id:0, flags:0 > ovs-ctl[13868]: * Starting ovs-vswitchd > ovs-ctl[13868]: * Enabling remote OVSDB managers > systemd[1]: Started Open vSwitch Internal Unit. > > P.S. adding a few people I usually see replying on these topics directly. > Thanks in advance for sharing your thoughts! > > Christian Ehrhardt > Software Engineer, Ubuntu Server > Canonical Ltd >